Retrieval and generation separately. Retrieval: recall@k and nDCG on labelled question-document pairs. Generation: faithfulness (every claim supported by the context), relevance and citations. Many bad answers are retrieval failures.
Card 1 of 292. Answer: Retrieval and generation separately. Retrieval: recall@k and nDCG on labelled question-document pairs. Generation: faithfulness (every claim supported by the context), relevance and citations. Many bad answers are retrieval failures.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Evaluate a RAG system in two stages: whether retrieval finds the passages that contain the answer, then whether the generated answer is faithful to them and on topic. Faithfulness means every claim in the answer is supported by the retrieved context, which differs from correctness against the truth.
Toy eval: Q1 has 2 relevant chunks, both in the top 5; Q2 has 1, ranked 7th; Q3 has 3, two in the top 5; Q4 has 1, ranked 1st. Recall@5 is 1, 0, 0.67 and 1, a mean of 0.67; the hit rate is 3/4. For generation, split an answer into claims and check each against the context: 3 of 4 supported is faithfulness 0.75, the RAGAS ratio (Es et al., 2023).
Interviewers ask where labels come from: real user questions with expert-marked passages beat LLM-written ones, which are easier. Label document spans, not chunk IDs, or re-chunking breaks the eval set, and spot-check LLM judges against humans. Watch crossed cases: an answer can be correct but unfaithful, from model memory, or faithful but wrong because the source is outdated.
