Contents
In brief
An engineer running a job platform shares what changed once a RAG system had to process more than 10,000 listings every day. Retrieval-augmented generation, or RAG, gives an LLM relevant retrieved passages before it produces an answer or score.
The difficult parts were not limited to the model. Document boundaries, input quality, vector storage, cost controls, and the ability to see silent failures determined whether the system stayed accurate and affordable under load.
What happened
The platform matches job listings to candidate profiles. A listing is first cleaned and normalized, split into passages, converted to embeddings, and searched by semantic similarity. An LLM then scores the relevance of each promising listing.
Fixed-size chunks performed poorly because they separated related requirements from their context. The author settled on recursive splitting: paragraphs first, then lines and sentences, with a small overlap between adjacent passages. That overlap preserves meaning when an important sentence falls at a boundary.
Input normalization was just as important. Applicant-tracking systems return descriptions in inconsistent formats and sometimes include HTML. Without a clean, consistent representation before splitting, empty or corrupted passages can reach retrieval without causing an obvious failure.
Why it matters
Chunking is not a late-stage tuning parameter. It determines what context retrieval can return, how many embeddings must be generated, and how many candidates reach the more expensive model-scoring stage.
The author compared OpenAI embeddings with a local model running on the same machine. The local option cost less, but it distinguished domain-specific requirements less reliably. The apparent saving at embedding time became noisier retrieval and extra LLM calls later in the pipeline.
Keeping vectors in pgvector alongside application data in PostgreSQL also simplified operations. A separate vector database can speed up a prototype, but it introduces synchronization work. In one database, a listing and its vectors can be written in one transaction rather than reconciled after a failure.
In practice
Start from the shape of your own documents instead of a generic chunk size. Job listings, contracts, and product descriptions have different structures, so their splitting rules should preserve different boundaries.
At volume, separate inexpensive retrieval from costly model evaluation. The author reduced spending by batching requests, caching results for similar candidate profiles, and using a less expensive model for common roles.
- Normalize text before chunking: remove HTML, standardize line endings, and validate required fields.
- Measure retrieved-passage quality against real queries, not only convenient development examples.
- Keep vectors with primary data when that simplifies transactions and backups; still test index settings and query time.
- Limit expensive LLM calls with batches, cached results, and model routing based on the accuracy each case needs.
- Trace each document through the pipeline and record passage counts, embedding time, errors, and the final score.
Observability should arrive before traffic does. In the reported system, correlated structured logs revealed that some sources produced empty passages: no job crashed, but their listings silently disappeared from search.
Takeaway
The account is a reminder that reliable RAG is mostly a data and operations problem. Model and index choices matter, but they cannot compensate for dirty inputs, broken passage boundaries, or a pipeline nobody can inspect.
Normalize data first, validate retrieval quality, control expensive model work, and observe the entire document path. With those foundations in place, scaling a successful demo is far less likely to become an opaque source of errors and spend.