Who said that? Why diarization is harder than transcription

Transcription has largely been solved for clean audio. Modern systems handle accented speech, technical vocabulary and moderate noise well enough that most people have stopped thinking about it.

Then you open the transcript of a four-person meeting and it is a wall of text with no indication of who said any of it. The information you actually wanted, that the objection came from the client rather than your own colleague, is absent.

That second problem is speaker diarization, and it has not been solved. On clean two-speaker audio it works well. On real conversational recordings the reported error rates sit around eleven percent, and on difficult real-world audio the state of the art is close to thirty-nine percent, which means roughly two out of every five seconds are attributed to the wrong person or missed entirely.

The gap between those numbers, and the gap between all of them and the figure a vendor will quote you, is the subject.

What diarization is, and what it is not

Three terms get conflated and they are different problems.

Speech recognition converts audio into words. What was said.

Diarization segments audio by speaker without knowing who those speakers are. It produces "speaker one spoke from 0:03 to 0:11, speaker two from 0:11 to 0:14". Who spoke when, with anonymous labels.

Speaker recognition matches a voice against known individuals. Who specifically, requiring enrolled voice profiles in advance.

Most products described as having speaker identification are doing diarization, then optionally attaching names by a separate mechanism, often just asking you. The distinction matters because diarization needs no prior knowledge of the participants and recognition needs a voice sample for each, which is a substantial operational difference and a meaningful privacy difference.

The word itself comes from diary: producing a record of who did what, when.

Why it is harder than transcription

Transcription has a helpful property that diarization lacks. Language is extraordinarily redundant, so a model that misses part of a word can usually recover it from context. Speaker identity has no such redundancy. Nothing in the sentence "I disagree" tells you who said it.

Four specific difficulties.

You do not know how many speakers there are. A transcription system knows it is producing words. A diarization system has to determine the number of distinct voices before or while assigning them, and getting that count wrong corrupts everything downstream. Splitting one person into two, or merging two into one, produces errors across the entire recording rather than in one place.

Voices are not stable. The same person sounds different when they raise their voice, move away from the microphone, become tired, or laugh. The variation within one speaker across an hour can exceed the variation between two different speakers, which makes the clustering problem ill-posed rather than merely difficult.

Turns are short. Conversation involves rapid exchanges, acknowledgements, interruptions. A half-second "mm-hmm" carries very little acoustic information about who produced it, and real conversation is full of them.

People talk over each other. This is the hard one and it gets its own section.

Overlap is the problem that broke the old approach

For most of the field's history, diarization worked as a pipeline of independent stages, and that architecture had a fatal assumption built into it.

Voice activity detection first, marking which regions contain speech at all and discarding silence, music and ambient noise. In a typical meeting recording this removes between thirty and sixty percent of the audio before anything else runs.

Segmentation next, cutting the speech into short chunks, usually one to two seconds, ideally at points where the speaker changes.

Embedding each chunk into a vector intended to capture voice characteristics rather than content.

Clustering those vectors into groups, one group per speaker.

This works, and the assumption that kills it is in the final step. Clustering assigns each segment to exactly one cluster, which means each moment of audio belongs to exactly one speaker. When two people talk simultaneously, the system must choose one and discard the other, and there is no configuration that fixes this because the limitation is structural.

Overlapping speech is not an edge case. In natural conversation it accounts for a substantial fraction of speaking time, concentrated exactly at the moments that carry the most information: interruptions, disagreements, corrections, the point where someone stops the speaker to object.

End-to-end neural diarization was the response. Rather than a pipeline of stages, a single network maps audio directly to per-speaker activity over time, framed as multi-label prediction so multiple speakers can be marked active in the same frame. Overlap is handled natively because nothing in the formulation requires a single answer per moment.

The refinement now common in production takes this further with powerset encoding: instead of predicting each speaker's activity independently, the model predicts which combination of speakers is active in each frame, including the empty set for silence and the various overlap combinations. This converts a multi-label problem into a single-label one over a larger label space, and models the dependency between speakers explicitly rather than treating them as independent events.

The pipeline, and why each stage inherits the last one's mistakes

Worth walking the stages once more with the failure modes attached, because the compounding is the part that surprises people building this for the first time.

Voice activity detection decides what is speech. Its errors are asymmetric in effect. A false negative, marking speech as silence, removes that audio from consideration entirely and no later stage can recover it. A false positive, marking noise as speech, injects a segment with no speaker in it, which then gets embedded and clustered like any other and pollutes whichever cluster it lands in. In noisy audio this stage alone can account for a large share of total error, which is why quoted figures using oracle voice activity detection are so much better than real ones.

Segmentation decides where to cut. Cut too long and a segment contains two speakers, so its embedding is a blend and belongs to neither cluster properly. Cut too short and there is insufficient acoustic material to characterise a voice, so the embedding is noisy. There is no length that is right for both problems, which is why fixed-window approaches trade one error for the other and multi-scale approaches exist to hedge.

Embedding converts a segment into a vector meant to capture voice rather than content. The failure here is subtle: embeddings trained to distinguish speakers on read speech may separate them less cleanly on conversational speech, where the same person's delivery varies far more. Recording conditions also leak into the embedding, so two people on the same microphone can appear more similar than one person across two microphones.

Clustering groups the vectors. Beyond the overlap limitation already described, this stage has to decide how many clusters exist, and that decision is made on the basis of geometry in embedding space rather than any knowledge of the conversation. A speaker who changes delivery mid-recording can split into two clusters; two speakers with similar voices can merge into one.

Each stage passes its errors forward and none can be corrected later. This is the structural argument for end-to-end approaches, independent of the overlap argument: not that any single stage is bad, but that the composition of four imperfect stages is worse than any of them.

The metric, and why your quoted number is flattered

Diarization is measured by diarization error rate, expressed as a percentage of speaking time attributed wrongly. It sums three failure types: speech attributed to the wrong speaker, speech missed entirely, and non-speech labelled as speech.

The number is straightforward. The conditions under which it was computed are where the trouble lives, because there are at least four standard ways to make a DER look better without improving anything.

Oracle speaker count. Telling the system in advance how many speakers there are removes one of the hardest sub-problems. Real deployments do not know this.

Oracle voice activity detection. Supplying ground-truth speech regions removes the errors that VAD would have made. Real deployments run their own VAD and inherit its mistakes.

A forgiveness collar. Excluding a small window around each speaker change from scoring, typically a quarter-second either side. Since transitions are where most errors occur, this removes a disproportionate share of them.

Excluding overlapped speech from scoring. Some historical reporting simply did not score the regions where people talk simultaneously, which is the hardest part.

A DER computed with all four is not comparable to one computed with none. The honest form, and the one careful papers now specify, scores all speech including overlap, with no oracle counting, no oracle VAD, no dataset-specific tuning and no collar. When comparing two systems, the reporting conditions matter more than the difference between the numbers.

One more distortion worth knowing: state-of-the-art results on a given benchmark are usually obtained by fine-tuning on that benchmark's training split. That is legitimate practice and it means the figure describes performance on audio resembling that dataset, not performance on yours.

The benchmark-to-reality gap

Published numbers cluster around a comfortable range, and the spread across datasets tells a story the headline figures do not.

On structured meeting corpora, error rates in the range of seven to twelve percent are current. Clean two-speaker telephone audio sits similarly. These are the numbers that appear in marketing.

On egocentric audio, recordings from wearable devices capturing everyday activity, the state of the art is around thirty-nine percent. Same task, same field, roughly four times the error.

The difference is not model quality. It is that meeting corpora are recorded with decent microphones, in rooms, with participants who mostly take turns, in a setting where people are aware they are being recorded. Everyday audio has movement, variable distance from the microphone, background activity, overlapping conversation that is not turn-taking, and speakers entering and leaving.

Widely used production systems report DERs in the eleven to nineteen percent range on standard benchmarks, and that spread within a single system across benchmarks is itself the point. The variance across conditions exceeds the variance across systems, which means choosing between vendors on benchmark scores is optimising the smaller term.

The practical consequence: benchmark performance predicts your performance only to the extent that your audio resembles the benchmark. For meeting recording it broadly does. For call-centre audio, field recordings, podcasts with remote guests, or anything captured on a phone in a room, it does not, and the only way to find out is to test on your own material.

What actually drives your accuracy

Three factors dominate, and model choice is not among them.

Microphone quality and placement. This matters more than almost anyone expects and it is the cheapest thing to fix. A single distant microphone capturing four people around a table is a hard problem. Individual close microphones make it nearly trivial, because separation is largely solved before any model runs. If the recording setup is under your control, changing it will outperform changing the model.

Number of speakers. Two is easy. Beyond about six, accuracy degrades noticeably, and the degradation compounds because speaker-count errors become more likely as the count rises.

Amount of overlap. Turn-taking conversation with clear boundaries is manageable. Argumentative or enthusiastic conversation, where people finish each other's sentences, is where systems fail, and it is exactly where the content is most worth attributing correctly.

There is a fourth option that sidesteps all of this. If you can record each speaker on a separate channel, diarization becomes trivial, because the assignment is known rather than inferred. Conference platforms increasingly expose per-participant audio, and where that is available the correct engineering decision is to use it and skip the problem entirely. A surprising number of teams run diarization on a mixed-down recording of a call whose platform could have given them separate tracks.

What to do about it

Test on your own audio before choosing anything. Twenty minutes of your actual recordings, hand-labelled, will tell you more than every benchmark in this article. The systems differ less from each other than your audio differs from their test sets.

Fix the recording before fixing the model. Better microphones, closer placement, per-speaker channels where possible. This is the highest-leverage intervention available and it is usually treated as out of scope.

Decide what an error costs you. Diarization for meeting summaries can tolerate a fair amount of error, since a summary that misattributes one comment is still useful. Diarization for legal transcription, medical records or anything where attribution is the point cannot. The acceptable DER is a function of what happens when it is wrong, not a general threshold. The same logic applies to evaluation generally.

Handle the overlap explicitly. If your audio contains substantial simultaneous speech, verify the system was scored on overlap rather than around it, and check whether it can output multiple speakers in a frame at all. Some cannot, architecturally.

Consider whether you need diarization or recognition. If the same participants recur, enrolling voice profiles and doing recognition may be both more accurate and more useful, at the cost of collecting voice samples, which has its own consent implications.

Where this is used, and what it is worth

The commercial demand sits in a few clear places, and the value differs sharply between them.

Meeting and call transcription is the largest by volume. Attribution turns a transcript into something searchable by participant and makes automatic summarisation useful, since a summary that cannot say who committed to what is much less useful than one that can. Tolerance for error here is relatively high.

Medical documentation separates clinician from patient in a consultation recording, which matters because the two carry different weight in a record. Error tolerance is low and the consequence of misattribution is a clinical record that says the wrong person reported a symptom.

Legal and compliance covers depositions, recorded calls under financial regulation, and evidence. Attribution is frequently the entire point, tolerance is very low, and the audio is often poor because it was not recorded for this purpose.

Media production uses it for subtitling, archive indexing and search across large recorded collections. Volume is high, tolerance is moderate, and the audio is usually good because it was professionally captured.

Research and analytics covers conversation analysis, call-centre quality monitoring and measuring who talks how much, which turns out to be a common request. Tolerance depends entirely on what is inferred from the numbers afterwards.

The pattern across these: the settings that most need accurate attribution frequently have the worst audio, because the recording was made for a different purpose or under no one's control. That inverse relationship is a large part of why the field's benchmark numbers and its practical reputation diverge.

What is unresolved

Whether end-to-end approaches will fully displace pipelines. End-to-end handles overlap natively and is conceptually cleaner. Clustering-based systems remain competitive on long recordings with many speakers, where the end-to-end formulation struggles to scale, and the strongest production systems are frequently hybrids. Whether this is a transitional state or a stable division of labour is not settled.

Handling an unbounded number of speakers. Most end-to-end formulations assume a maximum speaker count fixed at training time. Extensions exist for exceeding it, and none is clearly correct. For recordings with many participants or with people arriving and leaving, this remains awkward.

Whether DER is the right metric at all. It weights every second equally, which means a system that gets the substance right and misattributes filler scores identically to one that gets the filler right and misattributes the substance. Alternatives that weight by informativeness have been proposed and none has been adopted, partly because informativeness is hard to define and partly because a decade of published numbers uses DER.

Joint modelling with recognition. Diarization and transcription are usually separate systems joined afterwards, which discards useful information in both directions: who is speaking constrains what words are likely, and the words constrain who is speaking. Joint systems exist and have not clearly won, which suggests the interaction is harder to exploit than it appears.

The counter-argument

A piece emphasising how unsolved this is should acknowledge how solved it is for the common case.

For a clean recording of two to four people who mostly take turns, current systems work well enough that the failures are not the limiting factor in whatever you are building. That describes a large share of real demand: interviews, podcasts with good equipment, structured meetings. Teams in that situation should use a well-regarded system and stop thinking about it.

There is also a reasonable position that the hard cases are hard because the audio is bad, and that the effort is better spent on capture than on modelling. This is close to correct and it is not a criticism of the field, since some audio cannot be re-recorded. Historical archives, evidence recordings and field research all arrive as they are.

And the thirty-nine percent figure, while real, comes from a deliberately adversarial benchmark of everyday wearable audio. It is the honest upper bound on difficulty rather than a description of typical performance, and quoting it as the general state of the art would misrepresent things in the other direction.

The short version

Diarization answers who spoke when, which is a different and harder problem than transcription's what was said. Language is redundant enough that transcription recovers from partial failure; speaker identity has no such redundancy. The difficulties are not knowing the speaker count in advance, voices varying more within a person than between people, very short turns, and simultaneous speech.

The traditional pipeline of voice activity detection, segmentation, embedding and clustering carried a structural flaw: clustering assigns each moment to exactly one speaker, so overlap cannot be represented. End-to-end neural diarization replaced this by predicting per-speaker activity as a multi-label problem, and current systems refine it further by predicting which combination of speakers is active in each frame, which models speaker dependency explicitly.

Diarization error rate is the standard metric and the conditions under which it is computed matter more than the value. Oracle speaker counts, oracle voice activity detection, forgiveness collars around speaker changes, and excluding overlapped speech from scoring each flatter the number substantially, and state-of-the-art results are typically obtained by fine-tuning on the benchmark being reported.

Reported error rates run from about seven percent on clean meeting corpora to around thirty-nine percent on everyday wearable audio. That spread within the field exceeds the spread between systems, which means your audio's characteristics matter more than your vendor choice.

The practical conclusion is that the highest-leverage intervention is not the model. Better microphones, closer placement, and per-speaker channels where the platform offers them will outperform any model change, and a recording with separate tracks per participant removes the problem rather than solving it.

Common questions

What is speaker diarization? The task of determining who spoke when in an audio recording, producing segments labelled with anonymous speaker identifiers such as speaker one and speaker two. It answers who, separately from transcription which answers what. The two are usually run together and are distinct problems, and diarization is the substantially harder of the two on real audio.

What is the difference between diarization and speaker recognition? Diarization separates an audio stream by speaker without knowing who those speakers are, requiring no prior information about the participants. Speaker recognition matches voices against enrolled profiles of known individuals, requiring a voice sample for each in advance. Most products offering speaker identification perform diarization and then attach names by a separate route, frequently by asking the user. The distinction has real privacy consequences, since recognition requires collecting and storing voice biometrics and diarization does not.

Why is overlapping speech so hard? Because the traditional approach cannot represent it. Clustering assigns each audio segment to exactly one speaker cluster, so when two people speak simultaneously the system must pick one and lose the other, and no tuning fixes a limitation that is structural. End-to-end neural diarization addresses this by treating the problem as multi-label prediction, where several speakers can be marked active in the same frame. Overlap matters disproportionately because it concentrates at interruptions and disagreements, which is where the content usually is.

What is diarization error rate? The standard metric, expressed as the percentage of speaking time attributed incorrectly, summing three failure types: speech assigned to the wrong speaker, speech missed entirely, and non-speech marked as speech. The value is only meaningful alongside the conditions used to compute it, since oracle speaker counts, oracle voice activity detection, a forgiveness collar around speaker changes, and excluding overlapped speech all reduce it without any improvement to the system.

How accurate is speaker diarization in 2026? It depends far more on the audio than on the system. Clean meeting corpora and two-speaker telephone audio produce reported error rates around seven to twelve percent. Widely used production systems report roughly eleven to nineteen percent across standard benchmarks. Everyday audio captured on wearable devices, with movement, variable microphone distance and unstructured overlap, remains around thirty-nine percent at the state of the art. That spread across conditions is larger than the spread across systems.

How do I improve diarization accuracy? Fix the recording before the model. Better microphones and closer placement outperform any model change, and if your platform can provide separate audio channels per participant then the problem disappears, because assignment becomes known rather than inferred. Beyond that: reduce the number of simultaneous speakers where you can, and if the count is known in advance, supply it, since systems that accept a speaker count generally use it well.

Do I need diarization or per-channel recording? Per-channel recording, if you can get it. Diarization is an inference problem that exists because the speakers were mixed into one signal, and recording them separately removes the problem rather than solving it. Many conference platforms expose per-participant audio, and teams routinely run diarization on a mixed-down recording of a call whose platform would have supplied separate tracks. Check before building.

Which diarization system should I use? Test on your own audio rather than choosing from benchmarks, because the difference between your recordings and any test set will exceed the difference between the leading systems. Twenty minutes of your actual material, hand-labelled, is worth more than every published number. If your audio contains substantial overlap, verify the system can output multiple active speakers per frame and that its reported figures were scored on overlapped regions rather than around them.