Source Separation
Pulling one voice out of many — the cocktail party problem, named in 1953, and the solution reversed a decades-old assumption about how to represent audio.
When not to use it
- On heavily reverberant real rooms, expecting benchmark quality. The lab-to-meeting gap is large and under-reported.
- Spectrogram masking, for separation. The representation discards the phase the task needs.
- With unknown source counts. Much harder than the fixed-stem case, and nothing is clean.
- On arbitrary audio, expecting speech-level results. Voices have structure to exploit. Sounds don't.
Reach for something else instead
- Time-domain models (Conv-TasNet, Demucs) — what works. No representation loss.
- Multi-microphone / beamforming — spatial information makes it far easier. Use it if you have it.
- Speaker diarization — if you only need who spoke when, you may not need separation.
- Query-based extraction — pull out one named thing rather than splitting everything.
Sources & further reading
- Cherry (1953), Some Experiments on the Recognition of Speech, with One and with Two Ears — the cocktail party problem, named.
- Hershey et al. (2016), Deep Clustering: Discriminative Embeddings for Segmentation and Separation — permutation invariance done properly.
- Luo & Mesgarani (2019), Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation — beat the oracle, because the oracle was blind too.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming the ideal binary mask is a real ceiling. It's a ceiling on magnitude-only separation, and the phase you discarded is the point.
- Treating permutation invariance as a hack. There's genuinely no fact about which speaker is first.
- Benchmarking on dry mixes. Reverberation is the thing that breaks it.
- Optimising within the spectrogram. Sixty years did that, and the win came from leaving.
At a glance
FieldSpeech & Audio
NamedCherry, 1953
The reversaltime-domain models beat spectrogram masking, and beat its oracle ceiling
Whythe oracle only had magnitude; phase is what separation needs
The lessonthe hand-designed representation was the ceiling, invisibly
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Spectrogram masking vs. time-domain separation — one paints over regions in a representation that already threw away what you needed; the other keeps the waveform and beats the first one's theoretical ceiling.