Summary 06 — Retrieval Evaluation, Part II (Graded Relevance, BPREF, TREC & User-Based Evaluation)

Source: 04_Evaluation_IR_Part-II.pdf

1. Table of Contents (Topics Covered)

User-oriented measures: coverage & novelty ratios, relative recall, recall effort
Discounted Cumulated Gain (DCG) & graded relevance
The pooling method for large collections
BPREF and BPREF-10 (preference-based metrics)
Rank correlation metrics: Spearman & Kendall Tau
TREC conferences (collections, topics, relevant docs, tasks, measures)
Other reference collections (INEX, Reuters, OHSUMED, CLEF, etc.)
User-based evaluation: lab experiments, side-by-side panels, A/B testing, crowdsourcing, clickthrough data

Gist (2–3 paragraphs)

Standard precision/recall assume relevance is the same for all users and is binary. This lecture extends evaluation. User-oriented measures account for what a user already knows: coverage (fraction of known-relevant docs retrieved) and novelty (fraction of retrieved relevant docs that were new to the user), plus relative recall and recall effort. Graded relevance (highly vs. mildly relevant) is captured by Discounted Cumulated Gain (DCG), which rewards highly relevant documents appearing near the top.

For large collections it’s impossible to judge every document, so the pooling method judges only the top-K results pooled from many systems, treating unpooled documents as non-relevant. Because “not retrieved = non-relevant” is unreliable at scale, BPREF uses preference relations (is doc A preferred over doc B?), counting how many judged non-relevant docs precede each relevant one; BPREF-10 stabilizes it when few relevant docs are known. Rank correlation metrics (Spearman, Kendall Tau) compare how similarly two rankings order documents (coefficient from −1 to +1).

The TREC conferences operationalize all this at scale: yearly benchmarks with documents, topics (natural-language information requests), and pooled relevance judgments, supporting ad hoc and routing tasks plus many specialized tracks (QA, web, genomics, legal, spam…). Finally, because real preference depends on the user interface, Cranfield-style evaluation is complemented by user-based evaluation: controlled lab studies, side-by-side panels, A/B testing, crowdsourcing, and clickthrough data analysis.

2. Key Terminologies — Meaning & Use Cases

Term	Meaning	Use Case
Coverage ratio	Fraction of known-relevant docs that were retrieved	User-centric recall
Novelty ratio	Fraction of retrieved relevant docs new to the user	Measuring discovery
DCG	Graded-relevance metric discounting by rank position	Web/graded relevance
Graded relevance	Relevance on a scale, not just 0/1	Distinguishing highly vs. mildly relevant
Pooling method	Judge only top-K pooled results from many systems	Evaluating huge collections
BPREF	Counts judged non-relevant docs ranked before relevant ones	Incomplete judgments
BPREF-10	BPREF variant ensuring ≥10 preference relations	Few known relevant docs
Spearman coefficient	Rank correlation from position differences	Comparing two rankings
Kendall Tau	Rank correlation from concordant/discordant pairs	Comparing two rankings
TREC	Annual large-scale IR evaluation conference	Standard benchmarks
Topic	TREC’s natural-language information request (num/title/desc/narr)	Test queries
Ad hoc vs. routing task	New queries on fixed DB vs. fixed queries on changing DB	Search vs. filtering
A/B testing	Random subset sees new design; compare behavior	Live UI/ranking evaluation
Clickthrough data	Implicit relevance signal from user clicks	Cheap large-scale feedback
Crowdsourcing	Distributed human judgments (e.g., MTurk)	Scaling relevance judging

3. Process Workflow Diagram

flowchart TD
    A[Large Document Collection] --> B[Multiple IR systems run topics]
    B --> C[Pool top-K results]
    C --> D[Human assessors judge pool]
    D --> E[Relevance judgments
binary or graded]
    E --> F1[DCG: graded relevance]
    E --> F2[BPREF / BPREF-10: preferences]
    E --> F3[MAP, P@10]
    F1 --> G[Compare systems at TREC]
    F2 --> G
    F3 --> G
    H[Real users + UI] --> I[A/B test / panels / clickthrough]
    I --> J[User-based evaluation]
    G --> K[Decide best ranking]
    J --> K

4. ELI5 — Complex Terms Explained Simply

Coverage vs. novelty: Coverage = “did it find the ones I already knew about?” Novelty = “did it surprise me with new good ones?”
DCG: Like grading a playlist — a great song at track 1 counts more than the same song buried at track 50; weak songs near the top hurt the score.
Pooling: You can’t taste every dish at a giant buffet, so you only judge the plates that several chefs recommended, and assume the rest aren’t worth it.
BPREF: Instead of exact scores, just ask “is this good page ranked above the known-bad pages?” Counts how often bad stuff sneaks ahead of good stuff.
Spearman / Kendall Tau: Two judges rank the same contestants — these numbers say how much the two judges agree (+1 identical, −1 opposite).
TREC: The “Olympics” of search — everyone competes on the same documents and questions so results are comparable.
Topic: A carefully written description of what counts as a relevant answer, so human judges agree.
A/B testing: Show two versions to two random groups and let real behavior pick the winner.
Clickthrough data: Treat clicks as quiet votes — what people click is a hint about what’s relevant.