Self-Supervised Learning
Learning from unlabelled data by inventing the labels from the data itself — the idea that made every modern model possible.
When not to use it
- (You mostly won't do it. The question is when the framing misleads.)*
- Training it yourself. This is a frontier-lab activity. Download the result.
- With a pretext task that has a shortcut. The model will find it and learn nothing. Early vision work is a catalogue of this.
- Contrastive learning with small batches. You need enough negatives for the contrast to mean anything.
- Expecting it to escape the data constraint. It replaced a label bottleneck with a data bottleneck.
Reach for something else instead
- Transfer learning from a pretrained model — what you'll actually do.
- Supervised learning — if you have labels and a narrow task, this is simpler and often better.
- Weak supervision — noisy rules instead of annotations.
Sources & further reading
- Devlin et al. (2019), BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — masked prediction; the pretext task that reshaped NLP.
- Chen et al. (2020), A Simple Framework for Contrastive Learning of Visual Representations — SimCLR; contrastive learning and why batch size matters.
- He et al. (2022), Masked Autoencoders Are Scalable Vision Learners — MAE; mask 75%, which is far more than anyone expected and that's why it works.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating it as one technique among many. It's the thing every modern model is.
- Designing a pretext task with a shortcut, then wondering why the representations are useless.
- Reading LeCun's cake as a result. It's an intuition, and RL on verifiable rewards has been doing more than cherry duty lately.
- Assuming unlabelled data is unlimited. That's the current strategic problem.
At a glance
FieldDeep Learning
The trickhide part of the data, predict it; the label was already there
What it enabledevery modern model; GPT, BERT, CLIP are all this
Why pretext design mattersa shortcut means no learning
MAE's findingmask 75%, far more than expected
The new bottleneckdata, not labels
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Self-supervised vs. supervised learning — one invents its labels from the data and can use the whole internet; the other needs annotators and stops when the budget does.