Audio Classification
Naming what a sound is — where the field borrowed vision's entire playbook, including its label problems.
When not to use it
- For precise event timing, with AudioSet-pretrained models. The labels are clip-level. It knows whether, not when.
- Trained from scratch. Pretrain on AudioSet. Always.
- Without matching your acoustic environment. Mic, room and distance shift things harder than in vision.
- Assuming translation invariance on the frequency axis. Shifting up changes the pitch, which may change the class.
Reach for something else instead
- Self-supervised audio (wav2vec 2.0, HuBERT) — learned representations. Where this is going.
- CLAP — zero-shot by writing the class name. CLIP's trick for sound.
- Classical DSP — for narrow, well-characterised sounds, a filter still works and costs nothing.
- Multi-microphone — spatial information is a free feature people forget.
Sources & further reading
- Gemmeke et al. (2017), AudioSet: An ontology and human-labeled dataset for audio events — the ImageNet of audio, and its labels are weak by construction.
- Kong et al. (2020), PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition — the workhorse pretrained models.
- Gong, Chung & Glass (2021), AST: Audio Spectrogram Transformer — a ViT on spectrograms, and it needed ImageNet pretraining, which is revealing.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Deploying clip-level models for event timing. AudioSet's labels don't contain that information.
- Assuming a spectrogram is an image. One axis isn't translation-invariant, and the CNN's bias is half-wrong.
- Ignoring severe class imbalance. Speech and music dominate AudioSet enormously.
- Skipping SpecAugment. It's cutout from vision, and it's the biggest single win.
At a glance
FieldSpeech & Audio
The routeaudio became a vision problem; CNNs on spectrograms, transplanted whole
The datasetAudioSet, with clip-level weak labels
The odd factImageNet pretraining helps, though a CNN's bias is half-wrong for spectrograms
The patternaudio borrows everything, about two years late
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Spectrogram + CNN vs. self-supervised audio — one borrows vision's architecture and half its assumptions; the other learns a representation from the actual signal, and is winning for the obvious reason.