Key takeaways

A vector database is a derived index, not the system of record. Canonical bytes, versions, ACLs, and validity windows belong in source systems or an object store. The vector store holds a search projection and must be rebuildable from a manifest.

Approximate nearest neighbors is an explicit trade among recall, latency, and memory. HNSW, inverted-file (IVF) indexes, quantization, and disk graphs such as DiskANN buy different points on that surface. A number from someone else's benchmark, without your filters and embedding model, proves nothing.

Access and tenant predicates belong inside the index query, not after it. Fetch-then-drop leaks restricted text into traces, caches, and model context, and it also yields empty results when the matching neighbors sit outside the unfiltered top-k.

PostgreSQL with pgvector is a legitimate production default. It wins when vectors sit next to transactional data, memory is predictable, and you do not want a second operations plane. It loses when millions of vectors, harsh filters, quantization, or disk-resident graphs become the normal path rather than an experiment.

Changing the embedding model is a data migration. Dimension, distance metric, and model version are schema. Mixing spaces in one index is silent quality failure.

Deletes are a first-class contract. Tombstones, delayed compaction, and forgotten IDs in a graph are why a revoked policy still appears in answers.

Vector database vs relational vs search engine

A relational database answers “which rows satisfy a predicate.” It is strong at schema, joins, transactions, and exact keys: contract number, status, date, foreign key. A search engine answers “which documents best match this query text.” It is strong at lexicon: morphology, term statistics, BM25, highlighting, facets. A vector database answers a third question: which objects are closest to this vector under a chosen metric, usually with extra predicates on attributes.

Those are different physical operators. Nearest neighbor search in hundreds or thousands of dimensions does not collapse to a B-tree on one column, and it does not collapse to a postings list of terms. Hence graph links, inverted lists of clusters, quantized codes, and paged disk graphs.

In production the three layers coexist.

flowchart LR
  app[Application] --> q[Query]
  q --> v[Query vector]
  q --> f[ACL and attribute filters]
  v --> idx[Vector index]
  f --> idx
  q --> lex[Lexical index]
  f --> lex
  idx --> ids[Candidate IDs]
  lex --> ids
  ids --> canon[Canonical chunks]

The legal document stays in the source system or object storage. Relational tables hold facts and grants. The search engine holds text for identifiers and rare tokens. The vector index holds numeric projections and pointers. Collapse everything into one “magic” store and you either give up transactional guarantees, pay for semantic search where BM25 was enough, or park heavy originals in an engine that was not built as a CMS.

A dedicated vector database is justified as an index service when:

• vector count and query rate require specialized approximate search;
• tenant and payload filters must run during retrieval, not after;
• read/write scale must move independently of the OLTP core;
• the team will actually operate a second plane: backups, observability, reindex, memory quotas.

It is not justified by “put PDFs in a database and ask an LLM.” That failure is a missing ownership boundary: who owns the original, who computes vectors, who deletes stale rows.

How this index sits among ingestion, lexical search, reranking, and generation is the subject of enterprise RAG architecture. Here we stay on what the index promises.

Vectors and metrics

An embedding is a numeric image of a chunk or a query. The model maps text into a fixed-dimensional space. Closeness in that space correlates with semantic closeness on the distribution the model saw — it does not guarantee usefulness on your corpus. Why embeddings exist, and where they stop, is covered in why embeddings matter. Storage only cares about the consequences.

The index must know three facts and must not mix them quietly:

1. Dimension. A 384-d vector does not belong in a 1024-d index with zeros padded on the end.
2. Metric. Cosine, inner product, and Euclidean (L2) induce different rankings. A model trained for cosine should not be queried under L2 without evidence.
3. Model version. Two models with the same dimension and “cosine” still define incompatible spaces.

Normalized vectors make cosine and inner product nearly interchangeable; unnormalized vectors do not. If a vendor says “use inner product,” check whether they normalize outputs for you. A metric mismatch looks like “search got stupid after a client upgrade,” not like a process crash.

Do not store a bare vector without chunk ID, document ID, model version, validity time, and an access key. Those fields are how you filter, delete, and rebuild. You may cache chunk text in the payload for debugging; it must not be canonical. Otherwise a parser upgrade forks the truth.

Approximate nearest neighbors: HNSW, IVF, quantization, DiskANN — recall vs latency vs memory

Exact nearest neighbors over millions of high-dimensional vectors is a scan, or a structure that degrades toward a scan. Production indexes therefore run approximate nearest neighbor (ANN) search: they target high recall (the fraction of true neighbors recovered), not mathematical completeness. Recall, latency, and memory move together. Push one axis and the other two usually pay.

HNSW graphs

HNSW (Hierarchical Navigable Small World) builds a layered graph: long-range links on upper layers, dense local links at the base. Search descends from coarse to fine. The knobs teams actually turn:

• M — max connections per node. Higher M raises recall and memory, and slows inserts.
• efConstruction — search width at build time. Higher builds a better graph and a slower index job.
• efSearch (or equivalent) — search width at query time. This is the main recall-versus-latency lever without a rebuild.

Strengths: high recall when tuned, stable latency on unfiltered or lightly filtered queries, predictable behavior at moderate scale. Weaknesses: the graph wants RAM; inserts into a large graph are costlier than into a clustered IVF index; strict filters break connectivity — edges follow similarity, not “this tenant.”

Memory is roughly the vectors plus neighbor lists. Millions of float32 points with a non-trivial M already cost tens of gigabytes before payload. That is why quantization and disk layouts exist.

Inverted file (IVF) and relatives

IVF (often IVFFlat) partitions the space into lists, usually with k-means. The index stores centroids and the vectors in each list. A query inspects nprobe lists nearest to the query vector, not all lists. Knobs:

• nlist — number of lists. Too few: long lists and weak pruning. Too many: noisy clustering and a higher chance of missing the right list.
• nprobe — lists visited at query time. Direct recall/latency control.

Strengths: less graph overhead, an explainable story (“we scan 8 of 1024 lists”), a natural pairing with quantized codes inside a list. Weaknesses: recall tracks clustering quality; after heavy inserts the centroids go stale; skewed data inflates some lists.

IVF is a common choice when the set is large, RAM is tight, and the team will calibrate nprobe on its own set instead of copying a blog.

Quantization

Quantization compresses a vector: 8-bit scalar codes, product quantization (PQ), binary codes. The goal is to fit the index in memory or to compare codes faster than float32. The cost is lost discrimination among near neighbors. In practice, quantization is almost always paired with a short rerank over full vectors: coarse pass on codes, refine the candidate list.

The operational failure mode is turning on maximum compression because a table promised “16× savings,” then wondering why borderline queries collapsed. Compress what you measure. Keep full vectors if you intend to rerank; otherwise the memory win eats the last mile of quality.

Disk graphs: DiskANN and neighbors

When vectors and graph no longer fit in RAM, indexes aimed at SSDs appear: DiskANN (the Vamana graph plus page layout), on-disk modes in Milvus and Qdrant, and hybrids that keep a navigation skeleton in memory and fetch vectors in batches. The idea is the same: fewer random reads, sequential-ish page access, a warm page cache.

This is not free infinite capacity. Latency now depends on disk queues, fragmentation, and cache hit rate. Peak recall on a cold cache and on a warm cache are different numbers. A load test must include warmup and a share of unique queries, or you will measure the OS page cache rather than the index.

How to choose a structure

Do not start from the algorithm name. Write down:

• vector count and one-year growth;
• RAM budget per replica;
• p95 and p99 on queries that carry production filters;
• mutation mix (insert, upsert, delete) versus read;
• whether you have a hard filter such as one tenant among tens of thousands.

Rough, non-lawful ranges: up to a few hundred thousand vectors, exact search or a simple HNSW often suffices; millions in RAM — HNSW plus quantization or IVF; tens of millions with a RAM cap — a disk graph or a service that already operates one. Any range breaks on filters: an unfiltered index and tenant_id = X are different systems.

Third-party benchmarks are a map of capabilities and a bad way to pick a winner. Check dimension, metric, data distribution, fraction of filtered queries, and write mode. If they do not match, it is someone else's workload.

Filters, multi-tenancy, and hybrid search at the storage layer

Semantic closeness without a predicate is a lab mode. Almost every production query carries constraints: tenant, role, language, document type, validity, “approved only,” “not a draft.” The question is not whether filters exist. It is at which stage they run.

Pre-filter, post-filter, and predicate-aware search

Post-filter: take k neighbors, drop the ones that fail the predicate. On a rare predicate you get empty results even though matching vectors exist. On a sensitive predicate you also pull forbidden chunks through the retrieval layer.

Pre-filter: narrow IDs first, then search inside that set. On a tiny set this can be an exact scan. On a wide set you want an attribute index plus a vector pass.

Predicate-aware search: the engine uses payload indexes while walking the graph or the IVF lists, so it does not visit obviously forbidden nodes and does not throw away recall. This is what Qdrant (payload indexes), Weaviate (query filters), recent pgvector (iterative index scans), and k-NN plugins in OpenSearch and Elasticsearch sell. Implementation quality varies. Graph connectivity was built on similarity, not on tenant ID, so “filter while searching” is real engineering, not a checkbox.

Security rule: a forbidden chunk must not enter the application’s candidate list. Authorization is a deterministic predicate on a trusted side, not a prompt.

Tenant isolation

Three working models:

1. Collection (index) per tenant. Hard isolation, simple filters, a long operational tail: thousands of tiny indexes, uneven load, awkward quotas. Justified when data isolation is strict and you have relatively few large tenants.
2. Shared index plus a mandatory tenant_id predicate. Cheaper to run, easier to get wrong. You need payload indexes, an API that rejects unfiltered queries, and audit.
3. Physical partitions by tenant or tenant group. A middle path: fewer collections than tenants, better locality, harder rebalancing.

Noisy neighbors are literal: a hot tenant saturates graph-build queues and page cache. Write quotas, separate query pools, and per-tenant observability belong with the first paying tenant, not after the incident.

A data-subject deletion must reach the vector, the payload, and every replica — not only the canonical file. Otherwise the index is a shadow copy of personal data.

Hybrid search at the store

Hybrid search combines lexical and vector signals. Some products do it in one query: sparse plus dense in Qdrant, hybrid queries in Weaviate, tsvector beside pgvector in PostgreSQL, BM25 plus k-NN in OpenSearch/Elasticsearch. Rank fusion, including reciprocal-rank fusion, is covered in hybrid search with RRF. The storage question is narrower.

If the engine can return two ordered lists with stable IDs, you can fuse in the application. If you already pay for a vector database, ask whether the first pass should happen there: fewer round trips, one filter surface, one trace. If lexical search is already strong in an existing cluster, the vector service can stay a dense specialist and fusion can sit above it.

Do not fake hybrid by raising vector k. Identifiers, SKUs, error codes, and exact quotations belong to the lexical channel. The vector channel covers paraphrase and departmental synonymy. How you slice documents, which feeds both channels, is a separate discipline; see chunking experiments.

Product map: pgvector, Qdrant, Weaviate, Pinecone, Milvus, Chroma, kNN in OpenSearch/Elasticsearch

The table is not a ranking and not a substitute for a load test. It answers “which operating model you are buying.” Python clients, packaging, and the notebook-to-service gap are a different article: Python vector database libraries.

How to read it. If you already run a mature OpenSearch cluster and traffic is mostly lexical, adding a narrow k-NN channel there is often cheaper than standing up Milvus. If transactional truth already lives in PostgreSQL and semantic search is one product feature, pgvector cuts the number of systems. If payload filters and sparse-dense hybrid are the daily load, Qdrant or Weaviate sit closer to the job than “another OLTP column.” If a three-person team does not want to own an index, a managed service removes night pages and adds a bill plus a network dependency.

A note on “who is fastest” charts. They are sensitive to whether you count index build, warmup, filtering, quantization, client serialization, and payload fetch. A product that wins unfiltered search can lose on “this tenant, this language, documents newer than a year.” Measure your query.

Another hidden axis is the mutation model. Some systems are happiest with a nightly rebuild; others promise relatively fresh inserts into a live graph. If a policy changes at noon and the index catches up at 2 a.m., that is a product decision, not an efSearch tweak.

Operations: embedding version, reindex, deletes, cost

The index layer enters production after “insert works on staging.” Four plots break systems more often than HNSW versus IVF.

Model version as schema

Every point should carry a model_id (name, revision, dimension, metric). Query-time embedding must use the same version. Two models in one collection without isolation is silent space corruption: recall drops, “process up” stays green.

A model change is a new index or collection, dual writes during migration, recall comparison on a held-out set, then a read cutover. You cannot “backfill missing docs with the old model into the new index.” You cannot assume a cloud vendor will preserve the space when the model name changes. That contract is yours.

A practical pattern: chunk IDs stay stable; versioned projections live side by side (vec_v3, vec_v4) or in parallel collections. The application selects a projection explicitly. Rollback is a pointer change, not a midnight rebuild into the unknown.

Reindexing

A full rebuild is required when you change model, metric, index algorithm, payload schema, or chunking rules (the last one is already the RAG pipeline boundary). A partial rebuild is for a corrupt segment, mass deletes, or when IVF lists / quantization codes have drifted.

Reindex blue-green: the old index serves reads, the new one builds from canonical storage, you evaluate on the held-out set, you switch, you watch, only then you drop the old index. In-place rebuilds with an incompatible schema leave a window where queries see a mix of worlds.

Loader idempotence beats raw ingest speed: the same document at the same version must not spawn duplicate internal IDs. Duplicates steal top_k slots and fake recall.

Deletes and the right to be forgotten

Three levels teams confuse:

1. Hide from retrieval — a “do not show” filter. Fast, but vector and text remain.
2. Logical delete — a flag, exclusion from search, later physical compaction. An HNSW graph may keep tombstones for a long time; recall and latency degrade until compaction runs.
3. Physical delete — gone from disk and replicas. Required for personal data and for catastrophic ingest mistakes.

The delete contract spans canonical storage, every vector projection, caches, backups (with a policy, not “immediately everywhere”), and logs that may themselves contain the chunk. If the legal clock is “erase in 30 days,” an index that compacts in 45 is a process breach, not an engine footnote.

Test deletes the way you test inserts: a canary query that must go empty.

Cost

The vector layer bill is rarely one cloud line item. Add up:

• replica RAM (often the dominant term for unquantized HNSW);
• disk and IOPS for on-disk indexes;
• CPU for builds and quantization;
• re-embedding on model change (often more expensive than storage);
• network and capacity units for a managed service;
• human time for a second backup and observability plane.

A cheap managed index next to expensive re-embedding is false thrift. A heavy self-hosted cluster for a small, rare workload is the opposite mistake. Count cost per successful retrieval (the query that returned the right chunk inside the latency budget), not cost per gigabyte in a vacuum.

Minimum telemetry: percentile latency split by filtered vs unfiltered, recall on the held-out set after every index change, insert queue depth, delete rate, graph or list size, dimension-mismatch errors, and queries that omitted a mandatory tenant_id.

When PostgreSQL is enough and when you need a dedicated database

PostgreSQL with pgvector covers a surprisingly large production class: an internal assistant on tens or hundreds of thousands of chunks, vectors next to domain entities, transactional insert of “business row + vector,” a backup story the DB team already runs, SQL predicates auditors can read. Recent versions added HNSW, iterative scans under filters, and denser storage types. This is no longer the 2023 toy extension.

A dedicated database is warranted when several signals stack — not when a slide promises “billions of vectors.”

Decide with a load test on a copy of production mix: same filters, same empty-predicate rate, same payload size, same insert fraction. A synthetic “million random vectors, no filter” flatters the graph and lies to capacity planning.

Leaving PostgreSQL as system of record and moving only the search projection is usually cheaper than “migrate all documents into the new database,” which did not want the documents anyway. Dual-write during a recall bake-off is a controlled experiment.

The inverse mistake is starting a 3,000-chunk pilot on Pinecone or Milvus because a tutorial did. You buy network, invoice, and someone else’s ops rhythm before you know your predicate or your required recall.

Common mistakes

Treat the vector database as the only document store. The index grows a private truth: the parser changes, the payload does not. Canonical separately, projection separately.

Search without an access predicate, filter in the app. That is both a leak and a source of empty results. The predicate belongs in the index query and in the API contract.

Mix embedding models. Silent quality failure. Model version is a required field and a required query parameter.

Copy M, efSearch, nlist from a post. Those numbers were taken on another dimension, metric, and filter mix. Calibrate on your held-out set.

Max out quantization without a full-vector rerank. Paper memory savings, lost discrimination on near ties.

Ship Chroma in a container as “the production database.” A pleasant local loop is not backups, tenant isolation, or a growth plan. The notebook-to-service break is often client packaging, not HNSW — see the Python libraries note.

No delete path. A revoked policy, a mis-ingested contract, a data-subject request: without a physical path the index remembers.

Benchmark engines without filters. You will crown a lab winner and lose on day one with tenant_id.

Confuse index latency with RAG latency. The model hop, rerank, and context assembly are often slower than neighbor search. Optimize from a trace.

Grow by cloning collections by hand. Six months later nobody knows which index serves which product. You need a version manifest.

FAQ

Further reading

• Vector databases — short enterprise primer: why an index exists and how it inherits access control.
• Why embeddings matter — what vector representations encode, where semantic closeness fails, and why exact search remains.
• Production RAG engineering — the full pipeline. This pillar is storage and index only.
• Enterprise RAG architecture — platform view in which the vector index is one service among others.
• Hybrid search with BM25, vectors, and RRF — how to fuse lexical and dense channels.
• Chunking experiments — the fragment you store is a retrieval decision, not a character count.
• Python vector database libraries — client packaging and API stability, not neighbor microbenchmarks.

Conclusion

A vector database in production is an index with a contract, not a universal knowledge store. The contract covers metric and model version, approximate search with measured recall, tenant and ACL predicates inside the query, verifiable deletes, and a rebuild path from canonical storage.

PostgreSQL with pgvector remains an honest production starting point while volume, filters, and latency share an operations plane with transactional data. A dedicated product — Qdrant, Weaviate, Pinecone, Milvus, or a k-NN path in OpenSearch/Elasticsearch — is justified when that plane starts missing percentiles, memory, or isolation, not when it looks nicer on a slide.

Pick the neighbor algorithm after the query is clear: which predicate, which recall, which RAM budget, which mutation rhythm. Without those four numbers, a vendor bake-off is entertainment. With them, the index layer becomes a dull, operable part of the platform — and stops being the drawer where teams dump “all the RAG magic.”

In the Stuzhuk Lab sense, the same rule as the rest of the platform applies: first prove that the right chunk is found and is allowed to be found, and only then argue about how the model phrases the answer.