Every few months a new model ships with a longer context window, and teams rush to stuff their entire knowledge base into a single prompt. In production, that instinct is usually wrong.
The cost curve is the real constraint
Long-context inference is priced per token on the way in. A retrieval step that surfaces the right 4,000 tokens will almost always beat shipping 400,000 tokens of mostly-irrelevant context — both on latency and on dollars.
Relevance decay is real
Models attend unevenly across very long inputs. Facts buried in the middle of a 200k-token prompt get lost. A focused retrieval pass keeps the signal-to-noise ratio high.
What we do instead
Chunk and embed the corpus once, refresh on write.
Retrieve top-k, re-rank, and hand the agent a tight working set.
Reserve the long context window for genuinely long single documents, not for dumping the whole KB.
Bigger context windows are a capability, not a strategy. Retrieval is still how you ship reliable agents.
About the author Priya Raman, Head of Applied AI · Nova Labs
Priya Raman leads applied AI at Nova Labs, where she builds retrieval-augmented agent systems for enterprise customers.
