Why AI thinks hot and cold mean the same thing

Measure the similarity between the word vectors for "accept" and "reject" in a standard embedding model and you get roughly 0.73. For "long" and "short" it is about 0.71.

Those are high scores. On the same scale, related words like "dog" and "cat" sit around 0.8, and unrelated words like "dog" and "banana" sit near 0.2. Direct opposites are scoring almost as similar as synonyms.

This is not a defect in a particular model. It follows directly from the idea that makes embeddings work at all, and understanding why explains a set of failures that show up in search, retrieval, sentiment analysis and classification, usually without anyone connecting them to a common cause.

Embeddings encode which words appear in similar contexts, and opposites appear in almost identical contexts. "The water is hot" and "the water is cold" share every structural feature except the one that matters. The distributional signal that makes embeddings useful is the same signal that cannot distinguish a word from its opposite.

The idea underneath

Embeddings rest on a claim about language made decades before anyone could compute one.

Zellig Harris formalised it in 1954: words that occur in similar linguistic environments tend to have similar meanings. J.R. Firth put it memorably three years later, in the sentence that gets quoted whenever anyone explains embeddings: you shall know a word by the company it keeps.

The claim is stronger than it sounds. It says meaning can be recovered from distribution alone, without definitions, without reference to objects, without anyone explaining anything. Count enough co-occurrences and the structure of meaning falls out of the statistics.

It works. That it works is one of the more surprising empirical facts in computational linguistics, and everything downstream depends on it: word2vec, GloVe, sentence embeddings, vector search, retrieval-augmented generation, and the internal representations of every language model.

But look at what the claim actually licenses. It says words in similar contexts have related meanings. It does not say they have the same meaning, and it says nothing about the direction of the relation.

Why opposites collide

Consider the environments in which "hot" and "cold" appear.

The water is hot. The water is cold. It was a hot day. It was a cold day. Serve it hot. Serve it cold. Hot weather. Cold weather. Too hot. Too cold.

The syntactic frames are identical. The surrounding vocabulary is nearly identical. From a purely distributional standpoint, the two words are close to interchangeable, because the only thing distinguishing them is the aspect of the world being described, and that aspect never appears in the text.

This generalises. Antonym pairs are frequently more distributionally similar than unrelated words in the same domain, because opposites belong to the same semantic field, take the same grammatical roles, and get discussed in the same sentences. Accept and reject both appear near "application", "offer", "terms". Long and short both appear near "hair", "distance", "meeting".

The same mechanism produces a second confusion that is less discussed and equally consequential. Co-hyponyms, words that are siblings under a category, also collide: cat and dog, red and blue, Tuesday and Thursday. They appear in the same frames for the same reason. An embedding model reliably tells you that two words belong to the same conceptual neighbourhood and unreliably tells you which one you have.

The cosine similarity illusion

There is a related misunderstanding worth correcting, because it causes people to misread the numbers above.

Cosine similarity is defined on a range from -1 to 1, where -1 means the vectors point in exactly opposite directions. Many people assume that antonyms should therefore score near -1, and that a positive score indicates similarity.

In practice, cosine similarity between text embeddings almost never goes below zero. In high-dimensional spaces, two vectors pointing in opposite directions across hundreds of dimensions is vanishingly unlikely. Even where some components are negative, the sum across all dimensions stays positive. The effective range is roughly 0 to 1, and the theoretical range is a fact about the formula rather than about the data.

The practical consequence: a similarity score of 0.4 is not "mildly similar". It may be close to the floor. Anyone setting a threshold based on the intuition that 0 means unrelated has set it far too low, and the correct calibration has to be established empirically against the specific model and corpus, because different models produce different distributions.

This is one of the most common errors in production retrieval systems, and it produces the symptom of retrieval returning something plausible for every query including queries with no good answer.

The finding that complicates the story

Here is where it gets more interesting than the standard account.

The usual conclusion is that embeddings cannot distinguish antonyms from synonyms, full stop. That turns out to be wrong in an instructive way.

Research examining the geometry of embedding spaces has found that the information distinguishing synonyms from antonyms is present in the embeddings, and cosine similarity simply cannot see it. Applying a learned transformation, trained on a modest labelled sample, produces a space in which synonym distances shrink and antonym distances grow, and classifiers operating on the transformed representation separate the two reliably.

The distinction was always encoded. It is just not encoded along the axis that cosine similarity measures, which collapses a high-dimensional relationship into a single number about angle.

That reframes the problem. The failure is less about what embeddings represent and more about what a single similarity score can express. One number cannot simultaneously answer "are these about the same topic", "do these mean the same thing", and "do these point the same way", and cosine similarity has been asked to do all three.

What this breaks in practice

Four failure modes, all traceable to the same cause, and rarely diagnosed as related.

Semantic search returning opposites. A query about "increasing conversion" retrieves a document about decreasing it. Both are about conversion, both use the same vocabulary, and the retrieval layer has no mechanism for direction.

Retrieval systems that always return something. With a threshold set on the assumption that 0 means unrelated, every query clears the bar and the model receives context that is topically adjacent and factually wrong. This produces answers that read as grounded and are not, which is worse than an admission of ignorance.

Sentiment and classification drift. Systems relying on embedding proximity to seed terms inherit the confusion, since positive and negative terms in the same domain are distributionally close. A classifier trained on embedding features alone will find this boundary harder than the task appears.

Deduplication that merges contradictions. Similarity-based clustering groups a statement with its negation, because "the policy applies" and "the policy does not apply" are extremely close in embedding space. Negation is a single token and shifts the vector very little relative to the shared content.

That last one deserves emphasis. Distributional representations handle negation poorly, for the same structural reason as antonyms: "not" appears in every context, so it carries little distributional information, while the words around it carry a great deal.

A diagnostic you can run in ten minutes

The fastest way to find out whether this is affecting your system is to measure it directly, and the procedure is short enough that there is no excuse for reasoning about it instead.

Step one, find your floor. Take twenty queries with no answer in your corpus. Not hard queries, absent ones: ask about a topic your documents do not cover. Record the top similarity score for each. That distribution is your floor, and anything at or below it is noise the system is dressing up as a match.

Step two, find your ceiling. Take twenty queries with a known correct document. Record the score of the correct document. That distribution is your signal.

Step three, look at the overlap. If the two distributions overlap substantially, no threshold separates signal from noise and retrieval alone cannot be made reliable. This is the finding that matters and most teams have never looked.

Step four, test the direction problem specifically. Take ten queries and write the semantic opposite of each: increase becomes decrease, approve becomes reject, safe becomes dangerous. Embed both and measure their similarity to each other. If the pairs score above your noise floor, your retrieval layer cannot tell them apart and something downstream must.

Typical outcomes: floors much higher than expected, often 0.3 to 0.5 rather than near zero. Overlap between the distributions wider than anyone assumed. And opposite pairs scoring in the same range as correct matches.

None of this requires new infrastructure. It requires running your existing embedding model forty times and looking at the numbers, and it converts a vague suspicion that retrieval is unreliable into a specific measurement of how unreliable and where.

What actually helps

The fixes are known, unglamorous, and mostly not applied.

Calibrate the threshold empirically. Run a few hundred known-irrelevant queries against your corpus and look at the score distribution. Set the cutoff above the top of that distribution rather than at a number that felt reasonable. This is an afternoon of work and it fixes the always-returns-something failure outright.

Rerank with a cross-encoder. Embedding retrieval compares two vectors computed independently. A cross-encoder reads the query and the candidate together and scores the pair, which lets it notice that one says "increase" and the other says "decrease". Retrieve broadly with embeddings, then rerank narrowly. This is the single highest-return change available and it costs one extra model call per query.

Combine with keyword matching. Lexical search notices exact terms that embeddings smooth over. Hybrid retrieval catches cases where the specific word matters, which for negation and antonyms it usually does.

Retrieve fewer, not more. Accuracy in retrieval-augmented systems tends to peak at a handful of passages and degrade beyond, as noise accumulates. Widening the net to compensate for a poor similarity signal makes the problem worse.

Where the domain has known opposites, encode them. Counter-fitting post-processes an embedding space using an external lexical resource, pushing antonym pairs apart and pulling synonyms together. It works and it requires a curated list of the pairs that matter, which for a specialised domain is tractable and for open-domain text is not.

The deeper point: similarity is not one thing

Step back from the mechanics and there is a conceptual issue that the tooling obscures.

When two things are called similar, at least four different relations may be meant. They are about the same topic. They mean the same thing. One is an instance of the other. One is the opposite of the other, which is a form of relatedness rather than of distance.

Distributional methods measure the first well and conflate it with the rest. A single cosine score cannot distinguish them because it was never designed to; it measures proximity in a space built to capture contextual co-occurrence, and contextual co-occurrence is topic-relatedness.

This is worth holding onto beyond the antonym case, because it explains a whole class of surprise. When an embedding-based system does something inexplicable, the question to ask is which of those four relations the task actually required, and whether topic-relatedness was ever going to supply it.

What is unresolved

Whether contextual embeddings fix this. Modern sentence and document embeddings are contextual rather than static, so a word's representation depends on its sentence, which should help with polysemy and may help with negation. Whether it substantially addresses the antonym collision is not clearly established, and informal testing suggests improvement rather than resolution.

Whether the geometry finding generalises. The result that antonym information is recoverable via a learned transformation was demonstrated on specific models and datasets. Whether the same structure exists in every embedding space, and whether it can be extracted without labelled data, is open.

Whether it should be fixed in the embedding. One position holds that embeddings should encode direction, so antonyms end up far apart. The other holds that embeddings should encode topical proximity, which is what makes retrieval work, and directional distinctions belong in a downstream component. The second is winning in practice, largely because reranking is easy and respecializing an embedding space is not.

How much of this applies inside language models. The internal representations of a language model are not the same objects as a sentence-embedding model's outputs, and the extent to which model internals share this limitation is an interpretability question with no settled answer.

The counter-argument

The problem may be overstated for modern systems. Most of the striking numbers come from static word embeddings of the previous generation. Current systems use contextual embeddings, reranking and hybrid retrieval as standard, and a well-built pipeline already handles much of this. Presenting word2vec-era failures as current is a real risk in any article on this topic.

Topic-relatedness is frequently the right target. For search, grouping a query with documents about the same subject including opposing views is often exactly what a user wants. Someone searching "increasing conversion" may well want the article about what decreases it. The behaviour described here as a failure is sometimes the feature.

And the fix has a cost. Reranking adds latency and a model call per query, hybrid retrieval adds an index, counter-fitting adds a curation burden. For a system where topical retrieval is adequate, this apparatus is overhead, and adequate is more common than the failure cases suggest.

The short version

Word embeddings score "accept" and "reject" at roughly 0.73 similarity and "long" and "short" at 0.71, close to the 0.8 that related words like "dog" and "cat" reach. This follows from the distributional hypothesis, formalised by Harris in 1954 and phrased by Firth as knowing a word by the company it keeps: meaning is inferred from which contexts a word appears in. Opposites appear in nearly identical contexts, since "the water is hot" and "the water is cold" share every structural feature except the one being described, and that feature never appears in the text. Co-hyponyms such as cat and dog collide for the same reason.

A related misunderstanding compounds it. Cosine similarity is defined from -1 to 1 and in practice almost never goes below zero, because two high-dimensional vectors pointing in opposite directions are vanishingly rare. The effective range is roughly 0 to 1, so a score of 0.4 may be near the floor rather than mildly similar, and thresholds set on the assumption that 0 means unrelated are far too permissive.

The complication is that the information distinguishing synonyms from antonyms is present in the embedding geometry and cosine similarity cannot see it. A learned transformation on a modest labelled sample separates them reliably. The limitation is less about what embeddings encode and more about what a single similarity number can express, since one score is being asked to answer whether two things share a topic, share a meaning, and point the same direction.

Four practical failures trace to this: semantic search returning opposites, retrieval that always returns something because the threshold was set too low, sentiment systems inheriting the confusion, and deduplication merging a statement with its negation. The fixes are calibrating thresholds against known-irrelevant queries, reranking with a cross-encoder that reads query and candidate together, hybrid retrieval with keyword matching, retrieving fewer passages rather than more, and counter-fitting where the domain has a curatable list of opposites.

Common questions

Why do antonyms have similar word embeddings? Because embeddings encode which contexts a word appears in, and opposites appear in nearly identical contexts. "The water is hot" and "the water is cold" share the same syntactic frame and nearly the same surrounding vocabulary, and the only distinguishing feature is the aspect of the world being described, which never appears in the text. Antonym pairs are frequently more distributionally similar than unrelated words in the same domain, because opposites share a semantic field and take the same grammatical roles.

What is the distributional hypothesis? The claim that words occurring in similar linguistic environments tend to have similar meanings, formalised by Zellig Harris in 1954 and phrased by J.R. Firth as knowing a word by the company it keeps. It is the foundation of every word embedding, vector search system and language model representation. Its limit is built into its statement: it licenses the conclusion that words in similar contexts are related, not that they mean the same thing, and it says nothing about the direction of the relation.

Why is cosine similarity never negative in practice? Because in high-dimensional spaces, two vectors pointing in opposite directions across hundreds of dimensions are vanishingly rare. Even where individual components are negative, the sum across all dimensions stays positive. The theoretical range of -1 to 1 describes the formula rather than the data, and the effective range for text embeddings is roughly 0 to 1. A score of 0.4 may therefore be near the floor rather than mildly similar.

How do I set a similarity threshold for retrieval? Empirically, not by intuition. Run a few hundred queries you know have no good answer in your corpus, look at the resulting score distribution, and set the cutoff above the top of it. Thresholds chosen on the assumption that zero means unrelated are far too permissive, which produces the symptom of retrieval returning something plausible for every query, including queries with no correct answer.

Can embeddings tell synonyms from antonyms at all? Yes, though not through cosine similarity. Research on embedding geometry has found that the distinguishing information is present and encoded along axes that cosine similarity does not measure. A transformation learned on a modest labelled sample produces a space where synonym distances shrink and antonym distances grow, and classifiers on that representation separate them reliably. The distinction was always there; a single angle-based score cannot express it.

How do I stop semantic search returning the opposite of what I asked for? Rerank with a cross-encoder, which is the highest-return fix. Embedding retrieval compares two vectors computed independently, so nothing compares the query and candidate directly. A cross-encoder reads both together and can notice that one says increase and the other says decrease. Retrieve broadly with embeddings, then rerank narrowly. Adding keyword matching alongside helps too, since lexical search notices exact terms that embeddings smooth over.

Why do embeddings handle negation badly? For the same structural reason as antonyms. The word "not" appears across every context, so it carries very little distributional information, while the content words around it carry a great deal. "The policy applies" and "the policy does not apply" are therefore extremely close in embedding space, which causes similarity-based deduplication to merge a statement with its negation and causes retrieval to return contradictions as matches.

Does this problem still exist with modern embedding models? It is reduced and not eliminated. The most striking published figures come from static word embeddings of an earlier generation, and contextual embeddings, reranking and hybrid retrieval mitigate much of it in a well-built pipeline. The underlying cause has not gone away, because it follows from the distributional hypothesis rather than from any implementation detail, and any system inferring meaning from context alone inherits it to some degree.