Contents
In brief
Andrey Biryukov shows on Habr why a long context window does not replace preparing the text. Given 500 pages of network documentation, a model that received everything at once was right about a third of the time and also invented routes from addresses at the edges of the document. His move is to let a cheap model pull facts first, so the expensive model sees a short brief at the start and the single most important chunk at the end.
What happened
The occasion is familiar. Once context windows reached millions of tokens, it looked as if chunk search could be thrown out and the whole archive fed to the model. In practice attention holds the beginning and the end more tightly than the middle. Biryukov compares it to a student before an exam: the first and last pages stick, the middle blurs. The window is huge, and a question about a hundred-page document is answered worse than a smaller model with a short window would. He calls that a property of attention, not a random bug.
The check is concrete. Five hundred pages of network-equipment documentation — logs, addresses, firmware versions — are split into five parts of one hundred pages. Ten questions whose answers live only in the middle part. Full context on GPT-4o: about 34% accuracy, first token after about 14 seconds, plus invented routes stitched from addresses in the outer parts. Naive retrieval of the five nearest chunks lifted accuracy to about 42% and answered in a little over a second — fast, and still often wrong, because the link between chunks was gone. Growing chunks to ten thousand tokens, he writes, only moves the forgotten middle inside the chunk.
The “semantic gatekeeper” is a junior assistant beside an expensive specialist. The assistant is Mistral-7B-Instruct in 4-bit quantization, served with vLLM. A coarse vector search takes fifty chunks, not five. For each chunk the small model scores relevance from zero to one, extracts three facts, and answers in JSON, with no free-form retelling. Anything under 0.6 is dropped; the ten best of the rest are ordered by importance, not by document order. The start of the prompt gets compressed synopses of those ten chunks, about two thousand tokens. The end gets the single most relevant chunk in full, uncompressed. The important facts sit in both zones where attention is stronger.
His summary table, per thousand requests, looks like this. Full context: about 180 thousand tokens, 34.2% accuracy, 14.5 seconds, $180. Naive search: about 8 thousand tokens, 41.1%, 1.4 seconds, $15. RAPTOR: about 12 thousand tokens, 49.7%, 5.2 seconds, $40. Gatekeeper: about 9 thousand tokens, 68.4%, 3.8 seconds, $22. The Mistral-7B pass took 2.4 seconds, but the GPT-4 prompt became about twenty times shorter and saved about ten seconds. $22 versus $180 is the roughly eightfold saving on his prices.
Why it matters
A long window does not cure noise. A model that sees five hundred pages spends attention on irrelevant addresses and glues them into a plausible lie. A model that sees selected facts plus one full chunk at the end is wrong less often not because it got smarter, but because there is less to be distracted by. For documentation, contracts, and logs that matters more than the advertised size of the window.
These figures are one author’s run on ten questions into the middle of one corpus, not an independent benchmark. Read them as orders of magnitude: full context is expensive and weak in the middle, naive search is fast and blind to links between chunks, and a cheap fact-extraction step moved both accuracy and the bill. Do not carry 68% over to your archive without measuring again.
In practice
The scheme depends on forbidding invention at the cheap step. If the small model writes prose instead of extracting facts from the chunk, the errors land in the synopsis and the expensive model locks them in. The 0.6 cutoff and “ten best” are his settings, not a law. Measure on questions whose answer sits in the middle of the document: that is where a full window usually lies with the most confidence.
Duplicating the important chunk at the end is a deliberate trick for the attention curve, not a spare copy “just in case”. Cut both the opening synopsis and the full fragment at the end, and the trick falls apart.
- Do not feed the whole archive just because the window allows it. Check questions aimed at the middle.
- Separate fact extraction from the answer: the cheap model should return a score and facts as JSON, not a retelling.
- Over-fetch on the coarse search, drop weak chunks, and sort the rest by importance rather than page order.
- Put the compressed synopsis at the start of the prompt and the most important original chunk at the end.
- Count accuracy, tokens, time to first token, and money per thousand requests, as in his table.
Takeaway
A long context did not replace building the sample. On Biryukov’s corpus the full document lost to a scheme where a small model selects facts first and the large model reads a short prompt that shows the important part twice. It is a workable pattern for technical text if you repeat the measurement on your own questions instead of copying percentages from someone else’s table.
Comments