Context windows and what falls out of them
A large window is not a filing cabinet. What you put in it, and where, changes the answer more than most prompt wording does.
Context is your working memory per request, and it is rebuilt from scratch every time: the API is stateless, so a conversation is you resending the history. That is why a long chat gets slower and more expensive with every turn, and why an application that keeps appending will eventually spend most of its money re-reading itself.
Bigger windows tempt you into dumping everything in, and quality does not scale with quantity. Material in the middle of a long context is attended to less reliably than material at the ends, contradictory documents produce confidently blended answers, and irrelevant context measurably degrades accuracy. Curation beats capacity, which is the same lesson as Phase 00 on giving a model your codebase, with a bill attached.
The three levers when you run out are summarise, truncate and retrieve. Summarising preserves narrative and loses detail; truncation preserves recency and loses everything else; retrieval preserves relevance and introduces an entire subsystem you now have to evaluate. Lesson four of this phase is about the third one, and it exists because most teams reach for it before they have tried the first two.
You should now be able to
- Design what goes into a context window and in what order
- Recognise degradation before it looks like a model failure
- Choose between summarising, truncating and retrieving
Loading…