Small chunks retrieve precisely but strip the context that qualifies a fact; large ones keep context but dilute the embedding and spend tokens. Start near a few hundred tokens with some overlap, split on document structure, then tune against an eval set.
Card 1 of 292. Answer: Small chunks retrieve precisely but strip the context that qualifies a fact; large ones keep context but dilute the embedding and spend tokens. Start near a few hundred tokens with some overlap, split on document structure, then tune against an eval set.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
In RAG, chunk size trades retrieval precision against context: small chunks match a query sharply but lose the text that qualifies a fact, while large chunks keep context but blur the embedding and cost more tokens. A chunk is the unit a RAG pipeline embeds, indexes and returns whole.
Take a 10,000-token contract. At 200 tokens with 50 overlapping, the stride is 150, giving about 67 chunks; the top 5 add 1,000 tokens. One chunk may say "the fee is waived" while "For annual plans only" sits in the chunk before, so the model gets the rule without its condition. At 1,000 tokens there are 10 chunks, the top 5 are half the contract, and each vector averages several clauses.
The follow-up is how to get both: embed small chunks but return the parent section, prepend the title and heading to each chunk, and split on document structure. Many BERT-based embedders stop at 512 tokens, and libraries often truncate longer chunks silently. Compare sizes at a fixed token budget: top 10 small chunks against top 2 large ones.
