Dimensionality Reduction
Squashing many features into few — useful for compression and computation, and dangerous the moment you believe the picture.
When not to use it
- When you have enough data and compute. Reduction throws information away. If nothing forces it, don't.
- Before understanding your features. Reduce first and you've made your data uninterpretable before you learned what was in it.
- t-SNE/UMAP output as model input. They're visualisation techniques. The output isn't a metric space you can do arithmetic in.
- As evidence. A 2D plot showing clusters is a hypothesis. Test it in the original space.
Reach for something else instead
- Feature selection — pick a subset of real features. Keeps interpretability, which reduction destroys.
- Learned embeddings — reduction that knows what the task is.
- Regularisation — often the actual answer if the goal was reducing overfitting.
- Just using all the features — modern methods handle wide data better than the folklore suggests.
Sources & further reading
- van der Maaten & Hinton (2008), Visualizing Data using t-SNE — the original, and clearer than its reputation about what it does and doesn't preserve.
- Wattenberg, Viégas & Johnson (2016), How to Use t-SNE Effectively — the interactive piece showing how badly it can be misread. Essential.
- McInnes, Healy & Melville (2018), UMAP: Uniform Manifold Approximation and Projection — the current default, with a real theoretical grounding.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reading cluster sizes in a t-SNE plot. They carry no information.
- Reading distances between clusters in a t-SNE plot. Also no information.
- Not scaling before PCA. The largest-range feature becomes your first component and you've just measured units.
- Feeding t-SNE coordinates to a classifier. It's a picture, not a representation.
- Presenting a t-SNE plot as evidence of structure. It's a hypothesis. Perplexity will manufacture clusters in pure noise.
At a glance
FieldMachine Learning
Use PCAfor anything a model consumes
Use t-SNE/UMAPonly for looking
In a t-SNE plot, meaninglesscluster size, cluster distance
Rests onthe manifold hypothesis
Largely superseded bylearned embeddings
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
PCA vs. t-SNE — one is a linear, reversible transformation you can feed to a model; the other is a non-linear picture you should only look at.