Summary 11 — Latent Semantic Indexing (LSI): A Worked Example

Source: 08ex_Embedding_LSI.pdf

1. Table of Contents (Topics Covered)

A concrete 3-document LSI example (Grossman & Frieder)
Step-by-step LSI ranking procedure
Building the term-document & query matrices
SVD decomposition M = K·S·Dᵀ
Rank-2 approximation (dimensionality reduction)
Computing reduced document & query coordinates
Ranking by cosine similarity

Gist (2–3 paragraphs)

This short companion document walks through Latent Semantic Indexing on a tiny, concrete collection of three documents (d1: “Shipment of gold damaged in a fire,” d2: “Delivery of silver arrived in a silver truck,” d3: “Shipment of gold arrived in a truck”) with the query “gold silver truck.” It uses term frequency as weights, no stopword removal, no stemming, terms sorted alphabetically — keeping the math transparent.

The procedure: (1) build the term-document matrix M and query vector; (2) decompose M via SVD into K·S·Dᵀ; (3) apply a Rank-2 approximation, keeping the first two columns of K and D and the top 2×2 of S, reducing to a 2-dimensional concept space; (4) read off the reduced document coordinates from the rows of D (e.g., d1(-0.4945, 0.6492), d2(-0.6458, -0.7194), d3(-0.5817, 0.2469)); (5) project the query into the same space with q = qᵀ·Ks·Ss⁻¹; (6) rank documents by query-document cosine similarity.

The result: d2 ranks highest, then d3, then d1 — its reduced vector is closest to the query vector. This illustrates LSI’s key payoff: ranking emerges from latent concept relationships in the reduced space, not just literal term overlap, so the worked numbers make the abstract SVD machinery of the main Embedding lecture tangible.

2. Key Terminologies — Meaning & Use Cases

Term	Meaning	Use Case
Term-document matrix M	Rows = terms, columns = docs, entries = TF	Input to LSI
Query vector	Query encoded over the same terms	Becomes a pseudo-document
SVD (`M=K·S·Dᵀ`)	Factor matrix into orthogonal components	Reveal latent structure
K matrix	Term (left) eigenvectors	Term-concept mapping
D matrix	Document (right) eigenvectors	Doc coordinates in concept space
S (singular values)	Diagonal strengths of each concept	Decide what to keep
Rank-2 approximation	Keep top 2 dimensions	Reduce to concept space
Reduced query projection	`q = qᵀ·Ks·Ss⁻¹`	Place query in reduced space
Cosine similarity ranking	Order docs by closeness to query	Final LSI ranking

3. Process Workflow Diagram

flowchart TD
    A[3 documents + query
'gold silver truck'] --> B[Step 1: Build term-document matrix M
+ query vector]
    B --> C[Step 2: SVD → M = K·S·Dᵀ]
    C --> D[Step 3: Rank-2 approximation
keep top 2 dims of K, S, D]
    D --> E[Step 4: Doc coordinates
from rows of D]
    D --> F[Step 5: Project query
q = qᵀ·Ks·Ss⁻¹]
    E --> G[Step 6: Cosine similarity
query vs each doc]
    F --> G
    G --> H[Ranking: d2 > d3 > d1]

4. ELI5 — Complex Terms Explained Simply

The whole example: A mini-demo proving LSI works — three sentences, one search, and we watch the math pick the best match.
Term-document matrix: A spreadsheet counting which words appear in which sentences.
SVD: A blender that splits that spreadsheet into “what concepts exist,” “how strong each concept is,” and “how each sentence relates to those concepts.”
Rank-2 approximation: Throwing away the weak concepts and keeping just the two strongest — like summarizing a story in two themes.
Projecting the query: Putting your search onto the same simplified 2-theme map as the documents so they can be compared fairly.
Cosine similarity: Measuring which document arrow points the most similar direction to your query arrow — closest wins.
Why d2 wins: Even though all three docs share words, in concept-space d2 (“silver truck / delivery”) lines up best with “gold silver truck.”