Summary 05 — Retrieval Evaluation, Part I (Precision, Recall & Single-Value Metrics)

Source: 04_Evaluation_IR_Part-I.pdf

1. Table of Contents (Topics Covered)

Gist (2–3 paragraphs)

Evaluating an IR system means measuring how well it meets users’ information needs — essential for deciding whether a new ranking function should be launched or compared against BM25/vector baselines. Since relevance is subjective, evaluation uses reference collections (documents + information needs + human relevance judgments), an approach rooted in the Cranfield Paradigm (Cleverdon, 1950s), which gave us precision and recall.

Recall = fraction of all relevant documents that were retrieved; Precision = fraction of retrieved documents that are relevant. Because users see a ranked list, precision and recall change as they read down it, so results are plotted as a precision-recall curve at 11 standard recall levels (with interpolation), then averaged over many queries to compare algorithms. But P-R has limits: computing true recall requires knowing every relevant document, and a single number is often more practical.

So several single-value metrics exist. P@5/P@10 measure precision in the top 5/10 results (ideal for web search, where users want a few good hits, not high recall). MAP averages precision each time a new relevant document appears. R-Precision is precision at position R (the number of relevant docs). MRR rewards getting the first correct answer high (good for question-answering and known-item/URL queries). The E-measure lets the user weight precision vs. recall via a parameter b; setting b=1 gives the F-measure, the harmonic mean of precision and recall (high only when both are high).

2. Key Terminologies — Meaning & Use Cases

Term Meaning Use Case
Cranfield Paradigm Evaluating ranking with a fixed test collection Reproducible benchmarking
Reference collection Documents + information needs + relevance judgments Comparing systems fairly
Recall |R∩A|/|R| — relevant retrieved ÷ all relevant High-recall tasks (legal, medical)
Precision |R∩A|/|A| — relevant retrieved ÷ all retrieved Web search quality
Precision-Recall curve Precision plotted across recall levels Algorithm comparison
Interpolated precision Max precision at/above a recall level Smoothing P-R curves
P@5 / P@10 Precision in top 5 / 10 results Web search top-results quality
MAP Mean of average precisions across queries Standard IR benchmark score
R-Precision Precision at rank R (= #relevant) Per-query behavior
MRR Mean of 1/(rank of first correct answer) QA, known-item/URL search
E-measure Tunable precision/recall blend (param b) When one matters more
F-measure Harmonic mean of precision & recall Single balanced score

3. Process Workflow Diagram

flowchart TD
    A[Reference Collection
docs + topics + judgments] --> B[Run IR algorithm on test queries]
    B --> C[Ranked answer set A]
    C --> D[Compare A vs relevant set R]
    D --> E[Compute Precision & Recall
at each rank]
    E --> F[Interpolate to 11 recall levels]
    F --> G[Average over queries]
    D --> H[Single-value metrics]
    H --> H1[P@5 / P@10]
    H --> H2[MAP]
    H --> H3[R-Precision]
    H --> H4[MRR]
    H --> H5[F-measure]
    G --> I[Compare algorithms]
    H --> I

4. ELI5 — Complex Terms Explained Simply