Tokens cost money and latency, and accuracy sags for material buried mid-prompt. Retrieving the 2k tokens that matter usually beats pasting 200k. Long windows are for genuinely long documents; prompt caching makes a stable prefix cheaper, not free.
Card 1 of 292. Answer: Tokens cost money and latency, and accuracy sags for material buried mid-prompt. Retrieving the 2k tokens that matter usually beats pasting 200k. Long windows are for genuinely long documents; prompt caching makes a stable prefix cheaper, not free.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
A bigger context window is not always the answer: every extra token adds cost and latency, and models use facts buried mid-prompt less reliably than facts near the start or end. The context window is the maximum number of tokens, prompt plus output, a model handles per call.
Pasting a 200,000-token manual into every question sends 100 times the input of retrieving the 2,000 tokens that matter: at 10,000 questions a day, 2 billion input tokens versus 20 million. Time to first token grows with prompt length too. In "Lost in the Middle" (Liu et al., TACL 2024), GPT-3.5-Turbo given 20 documents did worse with the answer mid-context than with no documents at all.
Long context fits tasks that need the whole document, such as summarizing a contract or reasoning across a codebase. On prompt caching: a cached prefix skips recomputation and is billed at a discount, but new tokens still attend over it and it still fills the window. Passing a needle-in-a-haystack test, one literal lookup, does not prove a model reasons over all it holds.
