Source: 11_WebIR_Part-II.pdf
As the Web outgrew single-server engines, designs moved to massively parallel cluster-based architectures using document partitioning and replication across geographic locations (lower latency, fault tolerance). Variants include task- vs. data-parallel strategies, the distributed Harvest (gatherers + brokers), and multi-site architectures that answer local queries locally and route only externally-dependent queries elsewhere. Caching is critical: query distribution follows a power law, so caching results (and inverted lists) hugely boosts capacity; refinements include Probabilistic Driven Caching (prefetching) and knapsack-based static caching. Tiered indexes put frequent queries in a small fast index and the rest in a larger slow one.
Ranking is the hardest, most important function, drawing on three signal types: content (text/IR score, layout), structure (links, anchor text), and usage (clicks, geographic/temporal context). Link-based ranking exploits the insight that “good pages point to good pages”: PageRank models a random surfer and computes each page’s stationary probability (a Markov chain, with a damping/jump parameter q≈0.15, part of Google’s ranking); HITS computes mutually reinforcing hub and authority scores. Modern engines combine text (e.g., BM25) and link (PageRank) signals linearly (R = α·BM25 + (1−α)·PR), with α tuned (even per query type) via labeled or click data, and increasingly use Learning to Rank (pointwise/pairwise/listwise training) and neural ranking models. Evaluation uses P@5/10/20 (recall is unmeasurable on the Web) plus carefully debiased click-through data. A constant battle is Web spam (spamdexing: keyword stuffing, link farms, click spam, cloaking), distinct from legitimate SEO.
Managing Web data at scale needs special infrastructure: careful document-ID assignment (URL-sorted IDs aid compression), distributed stores (BigTable, HBase, Cassandra, Neo4j, MongoDB, ArangoDB, vector DB Milvus), Web graph compression (exploiting ~80% local links, ~3 bits/link), and duplicate detection via hashing (MD/SHA) and near-duplicate shingling. Finally, user interaction centers on the search rectangle paradigm, simple query languages with operators (+, -, site:, quotes), dynamic query suggestions (5× heavier than search; need spelling correction, diversity, freshness), the SERP layout (organic + sponsored results, title/snippet/URL, oneboxes, universal search), query assistance (“Did you mean” learned from query logs; recommendations via content-ignorant/aware/query-flow methods), faceted refinement, actionable results (translate, cached, stars), and the rise of zero-click search.
| Term | Meaning | Use Case |
|---|---|---|
| Cluster-based architecture | Partitioned + replicated index across machines | Web-scale serving |
| Document partitioning | Split corpus across index servers | Parallel query processing |
| Multi-site architecture | Geographically distributed query processing | Latency, scalability |
| Result/list caching | Store frequent answers/lists in memory | Huge throughput gains |
| Tiered index | Fast small + slow large index | Scalability |
| Ranking signals | Content, structure, usage | Computing relevance |
| PageRank | Random-surfer stationary probability (Markov) | Query-independent quality |
| Damping factor q | Random-jump probability (~0.15) | PageRank stability |
| HITS | Mutually reinforcing hubs & authorities | Topic-based link ranking |
| Anchor text | Link text describing target page | Structural signal |
| Learning to Rank | ML over pointwise/pairwise/listwise data | Modern ranking |
| P@5/10/20 | Precision at top positions | Web evaluation (recall unmeasurable) |
| Click-through data (debiased) | Implicit relevance from clicks | Scalable evaluation/training |
| Web spam / spamdexing | Deceptive ranking inflation (link farms, cloaking) | Adversarial IR |
| SEO | Legitimate page-description improvement | vs. malicious spam |
| BigTable / Milvus | Distributed store / vector database | Web-scale data, embeddings |
| Shingling | Near-duplicate detection via token sequences | Deduplication |
| Search rectangle / SERP | Search box / results page | User interaction |
| Dynamic query suggestions | Prefix-based as-you-type completions | Query assistance |
| Faceted search | Narrow results along dimensions | Query refinement |
| Zero-click search | Answer shown directly on SERP | Snippets, AI summaries |
flowchart TD
A[User Query in Search Rectangle] --> B[Dynamic Suggestions / Spelling]
B --> C[Distributed Cluster
partitioned + replicated index]
C --> D[Cache check
results + inverted lists]
D --> E[Candidate documents]
E --> F[Ranking]
F --> F1[Content signals: BM25/text]
F --> F2[Structure: PageRank, HITS, anchor text]
F --> F3[Usage: clicks, context]
F1 --> G[Combine: R = α·BM25 + 1-α·PR
+ Learning to Rank / neural]
F2 --> G
F3 --> G
G --> H[Spam filtering]
H --> I[SERP: organic + ads
title/snippet/URL, oneboxes]
I --> J[Query assistance / facets / actionable results]
J --> K[Click-through data] --> F