Do you actually need a vector database?

The standard architecture diagram for an AI product has a vector database in it. It's in every tutorial, every reference implementation, and every conference talk. So it goes in the plan, someone signs up for a managed service, and a week disappears into ingestion pipelines.

For a decent share of those projects, it was the wrong first move — and the tell is that nobody had checked whether retrieval helped before building the infrastructure to do it.

Here's how to work out whether you're one of them.

What it's actually for

A vector database does one job: given a chunk of meaning, find the stored chunks most similar to it, fast, across a very large collection.

That's it. It's a specialised index for similarity search. Text goes through an embedding model, comes out as a list of numbers, and the database finds other lists of numbers pointing in roughly the same direction. Because "roughly" is doing real work at scale — comparing against every stored item gets expensive fast — these systems use approximate algorithms that trade a little accuracy for a lot of speed.

Two things follow from that description, and both matter.

It's for scale. The approximation exists because exact comparison is too slow. If your collection is small, exact comparison isn't too slow, and the entire justification evaporates.

It's for semantics. It finds things that mean something similar. If what you need is exact matching, or filtering, or joining — that's a database you already have.

The number that decides it

Under roughly ten thousand documents, you probably don't need one.

At that size you can load your vectors into memory, compare against all of them with a few lines of NumPy, and get an answer in milliseconds. Not approximately — exactly. No service, no ingestion pipeline, no index tuning, nothing new to run at 3am.

The arithmetic is unromantic: a few thousand embeddings is a few tens of megabytes. That fits in RAM on anything. Brute-force cosine similarity across ten thousand vectors is a single matrix multiplication, and it's fast.

People find this suggestion undignified, which is precisely why it's worth making. There's a pull toward the architecture that looks serious. But an in-memory array is more accurate than an approximate index, has fewer failure modes, and costs nothing. The only thing it lacks is a logo on your diagram.

The cheaper thing you probably already have

Between "an array in memory" and "a dedicated vector database" sits an option most teams skip: your existing database can probably do this.

Postgres with pgvector stores vectors alongside your ordinary data and searches them with SQL you already know. Same for several others. The reason this matters isn't that it's slightly cheaper — it's that your vectors and your metadata live in one place.

That turns out to be the thing you actually want. Real queries are almost never "find similar text." They're "find similar text from this customer, in the last 90 days, excluding archived items." That's a similarity search with filters, and filters are what relational databases are for. Doing it across two systems means fetching candidates from one, filtering in application code, and discovering you filtered away everything you retrieved.

If you already run Postgres, try the extension before you add a service. The bar for a dedicated system should be "the extension wasn't fast enough," and you should have to say that sentence out loud with a number attached.

When you genuinely do need one

None of this means they're pointless. There are real cases, and they share a shape.

Millions of vectors. At this scale approximation stops being a compromise and becomes the point. Brute force isn't slow — it's impossible. This is what these systems were built for and they do it well.

Search is the product. If similarity search is the core of what you sell rather than a feature inside it, the tuning knobs matter and you'll want a system that exposes them. That's a legitimate reason.

High-throughput serving. Many concurrent queries, low latency, and the operational tooling to keep it up. Dedicated systems earn their keep here.

A team who'll operate it. This one's often decisive and rarely mentioned. A vector database is a stateful service. It needs monitoring, backups, upgrades, and someone to page when it stops. If you don't have that person, adding it isn't free — it's a debt with an unclear due date.

The mistake underneath the mistake

Here's the part that matters more than the tooling choice.

Most teams add a vector database as their first move. Before they've established that retrieval helps at all.

That's backwards. The question "does fetching relevant documents improve our answers?" is answerable in an afternoon with a script, a folder of text files, and no infrastructure whatsoever. Split the documents, embed them, do a brute-force search, put the top three in the prompt. If the answers get better, you've learned something real. If they don't, you just saved yourself a quarter.

And "they don't" happens more than the tutorials suggest. Sometimes the model already knows enough. Sometimes the documents don't contain the answers. Sometimes the failure is the question, not the knowledge.

Build the pipeline after you've proved the concept, not as the way to test it.

What you were actually buying

The uncomfortable finding, if you do run that afternoon experiment: the vector database is rarely the thing that decides quality.

When RAG systems disappoint, the cause is almost always upstream. The chunking split an answer across two pieces so no chunk contains it. The embedding model doesn't understand your domain vocabulary. The question is phrased in words the documents never use. Twenty passages got retrieved and the right one is buried in the middle of the context where the model barely attends to it.

None of those are database problems. All of them survive a migration to a faster index. You can swap vector stores and change nothing about your product, which is a strong hint about where the value actually sits.

The database is a lookup. The quality is in what you put in it and what you ask of it.

What actually costs you, and it isn't the licence

The pricing page is the least interesting cost here.

Ingestion is a pipeline, and pipelines rot. Documents have to be fetched, chunked, embedded and loaded — and then kept in sync as they change. That last part is where the work lives. A document gets edited and your index is now wrong, silently, until someone notices the answer is stale. Nobody budgets for reconciliation and everybody eventually builds it.

Re-embedding is a migration. Switch embedding models — for quality, for cost, because the provider deprecated yours — and every vector in the store is now meaningless. They live in a different space. You re-embed everything, which at scale is a job with a runbook rather than an afternoon.

Two stores means two truths. Your documents live in one system and their vectors in another, and they drift. A document gets deleted and its vector doesn't. Now your system confidently cites a passage that no longer exists, which is a worse failure than not finding it.

That last one is the strongest practical argument for keeping vectors next to your data rather than beside it. Not performance — consistency.

A decision you can make in five minutes

Do you have fewer than ~10,000 documents? Use an array. Brute force. Move on.

Do you already run Postgres? Try pgvector. Your filters and your vectors in one place is worth more than raw speed you won't notice.

Have you proved retrieval helps yet? If not, that's the actual next step, and it doesn't need any of this.

Do you have millions of vectors, real throughput, and someone to operate a service? Then yes — this is what they're for, and you'll get value from the tuning.

Is it in the plan because it's in every diagram? Take it out. Add it when something forces you to.

The general version

There's a pattern here that isn't about databases.

AI tooling has a strong default architecture, and the default is drawn for the hardest version of the problem. Most problems aren't that version. The diagram doesn't know your document count, your query volume, or whether retrieval helps at all — and it was drawn by someone with a product to sell or a talk to give.

The unglamorous version usually works: a script, some files, an array. Prove the idea, then buy the machinery the idea turns out to need.

Infrastructure you don't have can't break, can't cost, and can't be the thing you're debugging on a Friday.

The concepts behind this: vector databases, embeddings and RAG — each explained at five levels from plain English to the research frontier. If you're weighing retrieval against training, the decision tree takes about a minute.