It works in testing and fails on real data
The live data has drifted away from what the model was trained on, or the test set was contaminated by information the model would not have at prediction time.
Home/What went wrong
Troubleshooting index
Start from what is broken
Reference works are organised by the name of the thing, which assumes you already know what to call it. This is the same 256 concepts indexed the other way round: 34 things that actually go wrong, and what explains each one.
Models in production
Accuracy looks high but the model is useless
Accuracy is the wrong measure when classes are unbalanced. A model that always predicts the majority class can score well and predict nothing.
It was fine last quarter and is worse now
Nothing changed in the model. The world changed, and the relationship the model learned no longer holds.
The confidence scores do not mean anything
A model can be accurate and badly calibrated at the same time: its 90% is not really 90%.
It performs worse after I fine-tuned it on new data
Training on new data can overwrite what the model previously knew, particularly when the new set is narrow.
Training
Perfect on training data, poor on anything new
The model memorised rather than generalised. The classic failure, and the one most techniques exist to prevent.
Results look too good to be true
They usually are. Something in the training data encodes the answer, or the split leaked information between train and test.
Loss stops improving early in training
Gradients are dying before they reach the earlier layers, so those layers stop learning.
Training is unstable and the loss jumps around
Usually the step size, sometimes the batch size, occasionally unnormalised inputs.
I do not have enough labelled data
Labels are the expensive part. Several approaches sidestep needing more of them.
Quality degrades when I train on generated data
Training on model output compounds its errors across generations.
Language models
It states false things confidently
The same process produces true and false statements, and nothing internally distinguishes them.
It ignores instructions buried in a long prompt
Attention is unevenly distributed across a long context. Material in the middle is systematically underweighted.
It agrees with me even when I am wrong
Training on human preference rewards agreement, and the model has poor access to its own confidence.
Output is repetitive or wildly inconsistent
The sampling settings. Too deterministic loops, too random wanders.
It will not follow my output format
Asking in prose is unreliable. Constraining the output shape is not.
The same prompt gives different answers each time
Generation is stochastic by default. That is a setting, not a defect.
Answers are worse than they were with a shorter prompt
More context is not always better. Irrelevant material competes with the relevant material.
Retrieval and RAG
It retrieves the wrong documents
Most failures blamed on generation are retrieval failures. The model can only read what it was given.
It answers from its own knowledge instead of my documents
Nothing forces it to use the retrieved material, and nothing permits it to decline when the material is absent.
Chunks come back cut off mid-thought
The chunking strategy is splitting on length rather than meaning.
Search returns things that share words but not meaning
Keyword matching without semantic representation, or embeddings that do not suit the domain.
Agents
It loops forever without finishing
No decomposition of the goal, no reflection on progress, and no stopping condition.
It does something destructive I did not intend
Permissions exceeded what the task required, and no approval gate stood in front of an irreversible action.
It forgets what it did earlier in the task
Working state is not persisted between steps, and the context window is not memory.
Text in a document changed its behaviour
Instructions and data arrive in the same channel and the model cannot tell them apart.
It works in a demo and fails on real tasks
Demos sample the common case. Multi-step work compounds small error rates into large ones.
Cost and speed
It does not fit in memory
Weights, overhead, and a context cache that grows with use all have to fit at once.
Generation is too slow
Token generation is bound by memory bandwidth rather than raw computation.
Quality dropped after I quantized it
There is a floor. Below roughly four-bit, degradation accelerates sharply.
Costs scale badly with usage
Repeated context is being reprocessed, or a larger model is doing work a smaller one could.
Evaluation
It scores well on benchmarks and disappoints in use
The benchmark may be in the training data, or measuring something narrower than your task.
I cannot tell whether a change made things better
No fixed test set, so every comparison is against a moving target.
It performs differently for different groups of users
The training data under-represents some groups, and aggregate accuracy hides the gap.
Why this page exists. Nobody arrives at a reference work knowing the name of their problem. They arrive with a system that is behaving badly and a description of the behaviour. Every entry above maps a description to the concepts that explain it, and links to the full treatment where one exists.
Nothing here is new writing. It is the existing corpus with a second index over it, which is the only kind of index worth adding: one that matches how people actually search.