Do you actually need a vector database?

The standard architecture diagram for an AI product has a vector database in it. It's in every tutorial, every reference implementation, and every conference talk. So it goes in the plan, someone signs up for a managed service, and a week disappears into ingestion pipelines.

For a decent share of those projects. It was the wrong first move, and the tell is that nobody had checked whether retrieval helped before building the infrastructure to do it.

Here's how to work out whether you're one of them.

What it's actually for

A vector database does one job: given a chunk of meaning, find the stored chunks most similar to it, fast, across a very large collection.

That's it. It's a specialised index for similarity search. Text goes through an embedding model, comes out as a list of numbers, and the database finds other lists of numbers pointing in roughly the same direction. Because "roughly" is doing real work at scale, comparing against every stored item gets expensive fast, these systems use approximate algorithms that trade a little accuracy for a lot of speed.

Two things follow from that description, and both matter.

It's for scale. The approximation exists because exact comparison is too slow. If your collection is small, exact comparison isn't too slow, and the entire justification evaporates.

It's for semantics. It finds things that mean something similar. If what you need is exact matching, or filtering, or joining, that's a database you already have.

The number that decides it

Under roughly ten thousand documents, you probably don't need one.

At that size you can load your vectors into memory, compare against all of them with a few lines of NumPy, and get an answer in milliseconds. Not approximately, exactly. No service, no ingestion pipeline, no index tuning, nothing new to run at 3am.

The arithmetic is unromantic: a few thousand embeddings is a few tens of megabytes. That fits in RAM on anything. Brute-force cosine similarity across ten thousand vectors is a single matrix multiplication, and it's fast.

People find this suggestion undignified, which is precisely why it's worth making. There's a pull toward the architecture that looks serious. But an in-memory array is more accurate than an approximate index, has fewer failure modes, and costs nothing. The only thing it lacks is a logo on your diagram.

The cheaper thing you probably already have

Between "an array in memory" and "a dedicated vector database" sits an option most teams skip: your existing database can probably do this.

Postgres with pgvector stores vectors alongside your ordinary data and searches them with SQL you already know. Same for several others. The reason this matters isn't that it's slightly cheaper, it's that your vectors and your metadata live in one place.

That turns out to be the thing you actually want. Real queries are almost never "find similar text." They're "find similar text from this customer, in the last 90 days, excluding archived items." That's a similarity search with filters, and filters are what relational databases are for. Doing it across two systems means fetching candidates from one, filtering in application code, and discovering you filtered away everything you retrieved.

If you already run Postgres, try the extension before you add a service. The bar for a dedicated system should be "the extension wasn't fast enough," and you should have to say that sentence out loud with a number attached.

The three questions that settle it

Most of the debate disappears once you answer these honestly.

How many vectors, really? Not how many you hope to have. How many you have now, plus a realistic year. The answer is usually smaller than the initial estimate by an order of magnitude, because people count documents rather than chunks, or count the whole corpus rather than the part anyone will search. Below roughly a hundred thousand vectors, brute-force search in memory is fast enough that the question does not arise.

Does the index change while it is being queried? This is the one that actually distinguishes the tools. A static index rebuilt nightly is a file. An index taking writes continuously while serving reads, with deletes that must take effect immediately, is a database problem, and that is where a purpose-built system earns its keep. If your corpus updates in batches and can tolerate a rebuild, you need much less than you think.

Do you need filtering alongside similarity? Searching only within a tenant, a date range, a permission set. This sounds trivial and is the part that breaks naive implementations, because filtering after retrieval returns too few results while filtering before retrieval defeats the index. Systems that handle this properly are doing genuine work.

If you answered small, static and no, the tooling question is already resolved and it is not in favour of a new service.

What people are actually buying

It is worth being honest that the decision often is not technical.

A vector database is a legible line item. It appears in an architecture diagram, it has a vendor with documentation and a support contract, and it signals to whoever is reviewing the design that the retrieval problem has been taken seriously. An array of embeddings and a matrix multiplication in application code does the same job for many workloads and signals nothing at all.

There is a real argument on the other side of this, and it should not be dismissed. Choosing the boring managed option means someone else handles index maintenance, replication, and the awkward edge cases in filtered search that you would otherwise discover in production. Buying the operational burden away is a legitimate reason to buy something.

The failure mode is not choosing a vector database. It is choosing one before establishing whether retrieval quality is the bottleneck at all, which it frequently is not. Most retrieval systems that perform badly are failing at chunking, at embedding model choice, or at the absence of reranking, and none of those is fixed by changing where the vectors are stored.

When you do need one

None of this means they're pointless. There are real cases, and they share a shape.

Millions of vectors. At this scale approximation stops being a compromise and becomes the point. Brute force isn't slow, it's impossible. This is what these systems were built for and they do it well.

Search is the product. If similarity search is the core of what you sell rather than a feature inside it, the tuning knobs matter and you'll want a system that exposes them. That's a legitimate reason.

High-throughput serving. Many concurrent queries, low latency, and the operational tooling to keep it up. Dedicated systems earn their keep here.

A team who'll operate it. This one's often decisive and rarely mentioned. A vector database is a stateful service. It needs monitoring, backups, upgrades, and someone to page when it stops. If you don't have that person, adding it isn't free, it's a debt with an unclear due date.

The mistake underneath the mistake

The part that matters more than the tooling choice.

Most teams add a vector database as their first move. Before they've established that retrieval helps at all.

That's backwards. The question "does fetching relevant documents improve our answers?" is answerable in an afternoon with a script, a folder of text files, and no infrastructure whatsoever. Split the documents, embed them, do a brute-force search, put the top three in the prompt. If the answers get better, you've learned something real. If they don't, you just saved yourself a quarter.

And "they don't" happens more than the tutorials suggest. Sometimes the model already knows enough. Sometimes the documents don't contain the answers. Sometimes the failure is the question, not the knowledge.

Build the pipeline after you've proved the concept, not as the way to test it.

What you were actually buying

The uncomfortable finding, if you do run that afternoon experiment: the vector database is rarely the thing that decides quality.

When RAG systems disappoint, the cause is almost always upstream. The chunking split an answer across two pieces so no chunk contains it. The embedding model doesn't understand your domain vocabulary. The question is phrased in words the documents never use. Twenty passages got retrieved and the right one is buried in the middle of the context where the model barely attends to it.

None of those are database problems. All of them survive a migration to a faster index. You can swap vector stores and change nothing about your product, which is a strong hint about where the value actually sits.

The database is a lookup. The quality is in what you put in it and what you ask of it.

What actually costs you, and it isn't the licence

The pricing page is the least interesting cost here.

Ingestion is a pipeline, and pipelines rot. Documents have to be fetched, chunked, embedded and loaded, and then kept in sync as they change. That last part is where the work lives. A document gets edited and your index is now wrong, silently, until someone notices the answer is stale. Nobody budgets for reconciliation and everybody eventually builds it.

Re-embedding is a migration. Switch embedding models, for quality, for cost, because the provider deprecated yours, and every vector in the store is now meaningless. They live in a different space. You re-embed everything, which at scale is a job with a runbook rather than an afternoon.

Two stores means two truths. Your documents live in one system and their vectors in another, and they drift. A document gets deleted and its vector doesn't. Now your system confidently cites a passage that no longer exists, which is a worse failure than not finding it.

That last one is the strongest practical argument for keeping vectors next to your data rather than beside it. Not performance, consistency.

A decision you can make in five minutes

Do you have fewer than ~10,000 documents? Use an array. Brute force. Move on.

Do you already run Postgres? Try pgvector. Your filters and your vectors in one place is worth more than raw speed you won't notice.

Have you proved retrieval helps yet? If not, that's the actual next step, and it doesn't need any of this.

Do you have millions of vectors, real throughput, and someone to operate a service? Then yes. This is what they're for, and you'll get value from the tuning.

Is it in the plan because it's in every diagram? Take it out. Add it when something forces you to.

The general version

There's a pattern here that isn't about databases.

AI tooling has a strong default architecture, and the default is drawn for the hardest version of the problem. Most problems aren't that version. The diagram doesn't know your document count, your query volume, or whether retrieval helps at all, and it was drawn by someone with a product to sell or a talk to give.

The unglamorous version usually works: a script, some files, an array. Prove the idea, then buy the machinery the idea turns out to need.

Infrastructure you don't have can't break, can't cost, and can't be the thing you're debugging on a Friday.

The concepts behind this: vector databases, embeddings and RAG, each explained at five levels from plain English to the research frontier. If you're weighing retrieval against training, the decision tree takes about a minute.

The short version

A vector database stores embeddings and finds the most semantically similar ones to a query, which is the retrieval engine behind many RAG systems. But you do not always need a dedicated one. For small to moderate workloads, an in-memory vector library or a vector extension on a database you already run does the same job with far less operational overhead. A specialised vector database earns its place when scale, query volume, latency, or update frequency exceed what those simpler options handle comfortably, typically many millions of vectors or heavy traffic. The cost is not just hosting but the engineering and operational burden of running another system. The sensible path is to start simple and migrate when a concrete limit forces it. A vector database is retrieval infrastructure you should adopt when scale demands it, not by default, because most workloads start well below the point where a dedicated one earns its overhead.

Common questions

Do I always need a vector database for RAG? No. For a small or fixed corpus, up to roughly tens of thousands of chunks, an in-memory index or a vector-capable extension of a database you already run will serve, with far less operational overhead. A dedicated vector database earns its place when scale, update frequency, or metadata-filtered queries outgrow that.

What's the real cost of a vector database? Rarely the licence. The cost is operational: keeping the index in sync as documents change, tuning recall against latency, monitoring, and the engineering time all of that consumes. A tool that's free to install can still be expensive to run.

Can I start without one and add it later? Yes, and it's usually the right order. Begin with the simplest index that works, measure retrieval quality and scale, and adopt a dedicated vector database when a specific limit, corpus size, query latency, filtered search, actually bites. Premature infrastructure is its own failure mode.

What is a vector database? A vector database stores data as embeddings, numerical vectors that capture meaning, and is optimised to find the vectors most similar to a query vector quickly, even across millions of entries. This similarity search is what powers semantic retrieval: given a question turned into a vector, it returns the most semantically related chunks of stored text. Specialised vector databases add indexing, filtering, scaling, and persistence around this core operation. They are the retrieval engine behind many RAG systems, though for smaller workloads a library or a vector-enabled general database can do the same job.

When do you actually need a vector database? You need a dedicated vector database when your retrieval workload is large and demanding enough that simpler options struggle: many millions of vectors, high query volume, low-latency requirements, or frequent updates. Below that scale, an in-memory index or a vector extension on a database you already run is usually enough and far simpler to operate. The honest rule is to start with the simplest thing that works and add a specialised vector database when you hit a concrete limit, not because the architecture diagram calls for one. Premature adoption adds operational cost and complexity for scale you may never reach.

What are the alternatives to a dedicated vector database? Several options handle vector search without a standalone product. An in-memory vector library can index thousands to low millions of vectors inside your application with no separate service. Many general-purpose databases now offer vector extensions, letting you add similarity search to a database you already operate, which avoids running new infrastructure. Some full-text search engines also support vector search. For small corpora, even a straightforward similarity computation over stored embeddings can suffice. These alternatives cover a large fraction of real workloads, and moving to a dedicated vector database makes sense mainly when scale, latency, or update demands exceed them.

How much does a vector database cost? The cost is more than the hosting bill. There is the direct expense of the service or infrastructure, which scales with the number of vectors and query volume, but also the operational cost of running, monitoring, and maintaining another system, plus the engineering time to integrate and tune it. For large workloads these are justified. For small ones they are often disproportionate to the benefit, which is why starting with a simpler embedded or extension-based option and migrating later, once you have real scale and usage data, is usually the economical path.