Summary 18 — Neural Ranking Models

Source: 12_Neural_Ranking.pdf

1. Table of Contents (Topics Covered)

Gist (2–3 paragraphs)

Ranking is the core of IR. Traditional Learning To Rank (LTR) trains a model on hand-crafted features (e.g., BM25 as a query-document feature) — but defining and validating these features is domain-specific and laborious. Neural ranking models instead learn features and the ranking function end-to-end from raw text, overcoming hand-crafted limitations. Key challenges: queries are short while documents are long, query and document use different terms (so pure exact-matching fails — semantic matching is needed), and similarity is context-dependent. LTR has three training paradigms: pointwise (predict each pair’s exact relevance score), pairwise (predict which of two documents is more relevant — reduces ranking to binary classification), and listwise (operate on the whole ranked list, often optimizing the evaluation metric directly).

Neural ranking models fall into categories by how/when query and document interact. Representation-focused models (e.g., DSSM, often Siamese) encode query and document independently into vectors, then compare with cosine/MLP — efficient (document vectors are reusable) but risk missing fine matching signals. Interaction-focused models (e.g., DRMM with similarity histograms, DUET combining local+distributed networks) build query-document interactions early to capture matching signals — generally more effective but computationally heavier. Context-aware models (DeepRank) follow the query-centric assumption (relevance clusters around query-term matches). Attention-based models (ABEL-DRMM) weight tokens by attention. Others incorporate external knowledge bases/graphs (AttR-Duet) or pseudo relevance feedback (NPRF).

The state of the art uses deep contextualized language models, especially BERT: the query and document tokens are concatenated as [CLS] q [SEP] sd [SEP], and the [CLS] hidden state feeds an MLP to predict relevance; BERT’s Transformer layers can also serve as a contextual embedding layer or a semantic matching component. Nine recurring features characterize these models (symmetric, attention, ordered tokens, representation, interaction, early/late context injection, exact matching, external KB, deep LM). The neural ranking idea generalizes beyond document retrieval to structured-document retrieval, question answering, image retrieval (text-based and content-based/query-by-example), ad-hoc video search (concept- vs. embedding-based), and audio retrieval — all emphasizing semantic matching, often via cosine similarity over learned embeddings.

2. Key Terminologies — Meaning & Use Cases

Term Meaning Use Case
Learning To Rank (LTR) ML that learns to order documents Modern ranking
Hand-crafted features Manually engineered ranking signals Traditional LTR
Pointwise / pairwise / listwise Score each / compare pairs / order whole list LTR training paradigms
Semantic matching Match meaning despite vocabulary mismatch Beyond exact matching
Representation-focused Encode query & doc separately, then compare Efficient retrieval (DSSM)
Siamese architecture Shared network for both inputs Symmetric encoding
Interaction-focused Build query-doc interactions early Stronger matching (DRMM/DUET)
DRMM Histogram-based interaction matching Term matching signals
DUET Combines local + distributed networks Hybrid ranking
Query-centric assumption Relevance clusters around query terms Context-aware (DeepRank)
Attention mechanism Weight important tokens ABEL-DRMM, Transformers
Pseudo Relevance Feedback (PRF) Use top results as feedback NPRF
BERT Deep contextual Transformer LM State-of-the-art ranking
[CLS]/[SEP] tokens Classification / separator tokens BERT sentence-pair input
Cross-modal matching Match across text/image/video/audio Multimedia retrieval

3. Process Workflow Diagram

flowchart TD
    A[User Query] --> B[Traditional Ranking
BM25 → candidate docs]
    B --> C[Neural Ranking Model]
    C --> D{Feature extraction F}
    D -->|Separate encoders| E[Representation-focused
DSSM / Siamese]
    D -->|Early interaction| F[Interaction-focused
DRMM / DUET]
    D -->|Context| G[Context-aware / Attention
DeepRank / ABEL-DRMM]
    D -->|Contextual LM| H[BERT embeddings / semantic match]
    E --> I[Ranking model M
cosine / MLP]
    F --> I
    G --> I
    H --> I
    I --> J[Relevance score]
    J --> K[Train via LTR loss
pointwise/pairwise/listwise]
    J --> L[Ranked list to user]

4. ELI5 — Complex Terms Explained Simply