Variational Autoencoder
An autoencoder whose latent space you can actually sample from — the principled generative model that lost to GANs on looks and won by being useful.
When not to use it
- When sharpness is the product. Blurriness is structural under MSE. Diffusion is the answer now.
- When you only need compression. A plain autoencoder is simpler and reconstructs better; the KL term costs you fidelity you didn't need.
- With a very powerful decoder, carelessly. Posterior collapse means the latent gets ignored and you won't get an error.
- When a pretrained embedding exists. Usually better than a latent space you trained yourself.
Reach for something else instead
- Diffusion models — better generation, and they use a VAE internally anyway.
- Plain autoencoder — for pure compression, sharper and simpler.
- GAN — sharper, unstable, largely superseded.
- VQ-VAE — discrete latents, so a transformer can model them. The bridge to token-based generation.
Sources & further reading
- Kingma & Welling (2013), Auto-Encoding Variational Bayes — the paper; the ELBO and the reparameterisation trick.
- Higgins et al. (2017), β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework — the disentanglement dial, and its cost.
- van den Oord, Vinyals & Kavukcuoglu (2017), Neural Discrete Representation Learning — VQ-VAE; discrete latents, and the bridge to token-based generation.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Trying to tune away the blurriness. It's the MSE objective averaging plausible outputs. Change the loss, not the learning rate.
- Not watching for posterior collapse. KL going to zero looks like convergence and means the latent was abandoned.
- Treating β as free. It trades latent structure against reconstruction, directly.
- Using a VAE for compression when an autoencoder would reconstruct better.
At a glance
FieldGenerative AI
The changeencode to a distribution, not a point
The two lossesreconstruction + KL
Why blurryMSE averages plausible outputs
The trickreparameterisation, making sampling differentiable
Where it lives nowinside every latent diffusion model
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
VAE vs. GAN — principled and stable versus sharp and temperamental. GANs won the demos; VAEs are inside the diffusion models that beat both.