The context window
Everything the model can take into account at once, and why the middle of a long document is the most fragile part.
ObjectiveExplain what the context window is and apply three rules to place the useful information in it.
The model sees only what is in its context window: your instructions, the conversation history, the attached documents and the tool results. Its size is measured in tokens. Beyond it, you have to split, summarise or search.
A larger window does not mean even reading. A 2024 study measured that models retrieve information placed at the beginning or end of a long context better than information in the middle. The authors called this “lost in the middle”.
Three practical rules
- Put the instruction and the question at the end, after the documents, so they are fresh when the model answers.
- Attach the useful passages rather than a whole folder. Less noise, less forgetting.
- In a long conversation, ask for a summary of where things stand, then continue from that summary.
References
- Liu, Lin, Hewitt et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the ACL 12, p. 157–173. doi.org/10.1162/tacl_a_00638