Spectrogram
Turning sound into a picture so a vision model can look at it — the representation nearly all audio AI runs on, and it throws half the signal away.
When not to use it
- For separation or generation, uncritically. Phase is discarded and that's where it matters. Time-domain models beat spectrograms here.
- Expecting to invert one cleanly. You threw the phase away. Griffin-Lim guesses, and it sounds like a guess.
- MFCCs, in new work. Deep learning made them obsolete. It's inherited habit from the Gaussian mixture era.
- With a window chosen by feel. It's a physical trade between time and frequency resolution, not a knob.
Reach for something else instead
- Raw waveform — learn the frontend. Wins on separation; more compute.
- Learnable filterbanks — a middle path; the mel scale as an initialisation rather than a law.
- Complex spectrograms — keep the phase. Harder to model, and it's there.
- Self-supervised audio representations — wav2vec-style. What most modern systems actually use.
Sources & further reading
- Stevens, Volkmann & Newman (1937), A Scale for the Measurement of the Psychological Magnitude Pitch — the mel scale; 1937 psychoacoustics under modern AI.
- Griffin & Lim (1984), Signal Estimation from Modified Short-Time Fourier Transform — guessing the phase you threw away, and why it sounds metallic.
- Luo & Mesgarani (2019), Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation — time domain beat spectrograms, because phase.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Not realising phase is gone. It's the invisible lossy step and it's why generation is hard.
- Using MFCCs because tutorials do. They compress for a model class nobody uses anymore.
- Tuning window length for accuracy. You're trading time resolution for frequency resolution — that's physics.
- Skipping the log. Without it the representation is nearly all dark and models train badly.
At a glance
FieldSpeech & Audio
What it doessound becomes an image; every vision trick applies
The scalemel, from 1937 psychoacoustics, because it works
The invisible lossphase is discarded at the magnitude step
The hard constrainttime-frequency uncertainty; you can't resolve both
Winning on analysis, losing on generation
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Spectrogram vs. raw waveform — one is a hand-designed compression encoding real psychoacoustics and wins on analysis; the other keeps everything and wins where phase matters.