Speaker Diarization
Working out who spoke when — the unglamorous half of transcription, and usually the half that's wrong.
When not to use it
- When you can separate channels instead. Per-speaker microphones make the problem vanish. This beats any model.
- When the audio is heavily overlapping. Clustering-based systems assume one voice at a time and will produce confident nonsense.
- When attribution carries real consequence and nothing is reviewed. Assigning a commitment to the wrong person is worse than not knowing who spoke.
- When there's only one speaker. People run diarization on single-speaker audio and get spurious speaker splits.
Reach for something else instead
- Multi-channel recording — one mic per person. The actual fix.
- Speaker enrolment — provide voice profiles in advance, turning clustering into classification.
- Supplying the speaker count — removes the hardest guess if you know it.
- Manual attribution — for short, high-stakes recordings, a person is still better.
Sources & further reading
- Park et al. (2022), A Review of Speaker Diarization: Recent Advances with Deep Learning — the survey to read; covers the pipeline and its failure modes properly.
- Fujita et al. (2019), End-to-End Neural Speaker Diarization with Permutation-Free Objectives — EEND; handling overlap natively instead of assuming it away.
- Bredin et al. (2020), pyannote.audio: Neural Building Blocks for Speaker Diarization — the open toolkit most practical work starts from.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Letting the system guess the speaker count when you know it. Free accuracy, routinely left on the table.
- Reading a reported DER without checking whether overlap was excluded and a collar applied. That's where the errors live.
- Assuming good transcription implies good attribution. They're separate systems and the second is worse.
- Running diarization on single-speaker audio and getting phantom speakers.
- Not considering that you're generating biometric voice profiles as a side effect of taking notes.
At a glance
FieldSpeech & Audio
The questionwho spoke when
Metricdiarization error rate
Weak linkclustering, which must guess the speaker count
Breaks onoverlapping speech
Real fixseparate channels
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Diarization vs. speaker identification — diarization separates unknown voices into groups; identification matches a voice to a known person. One clusters, the other classifies.