Source: 08_Embedding.pdf
Embedding encodes words or documents as vectors in a vector space. The traditional bag-of-words approach yields sparse, long vectors (one dimension per vocabulary word, mostly zeros) that ignore context and treat synonyms (car/automobile) as unrelated. Modern word embeddings are short, dense vectors (typically 50–300 dimensions) of real numbers, learned unsupervised from large corpora — fewer parameters (helping generalization) and better at capturing synonymy and semantic similarity.
Latent Semantic Indexing (LSI) is an early embedding method that maps documents and queries into a space of concepts rather than terms. It applies Singular Value Decomposition (SVD) to factor the term-document matrix M = K·S·Dᵀ, then keeps only the s largest singular values for dimensionality reduction, filtering noise while preserving structure. A query is treated as a pseudo-document; cosine similarity in the reduced concept space ranks documents — so a document sharing concepts with a relevant one can be retrieved even without exact term overlap.
Word2Vec learns static, low-dimensional word vectors using a shallow neural network trained by self-supervision: nearby words in running text serve as “correct answers,” needing no hand-labeling. Two architectures exist: CBOW predicts a center word from its surrounding context (fast, good for frequent words), and Skip-gram predicts the context from a center word (better for rare words and small data). Because the softmax over the whole vocabulary is expensive, negative sampling trains a logistic classifier to distinguish true context words from a few randomly sampled negatives, updating only a small slice of weights per example. Newer dynamic/contextual embeddings (BERT) give a word a different vector in each context.
| Term | Meaning | Use Case |
|---|---|---|
| Embedding | Encode words/docs as vectors | Foundation of modern NLP/IR |
| Bag-of-words | Sparse vector of term presence/counts | Classic representation |
| Sparse vs. dense vector | Mostly-zero long vector vs. short real-valued | Efficiency & semantics |
| Synonymy capture | Similar words get nearby vectors | Better matching |
| LSI / LSA | Concept-space retrieval via SVD | Semantic retrieval |
| SVD | Factor M = K·S·Dᵀ |
Dimensionality reduction |
| Dimensionality reduction (s) | Keep top-s singular values | Filter noise, find concepts |
| Pseudo-document query | Query embedded into the doc space | LSI ranking |
| Word2Vec | Neural static word embeddings | Word similarity, features |
| Self-supervision | Use running text as implicit labels | No manual labeling |
| One-hot vector | Single 1 marking a word’s index | Network input |
| Softmax | Turns scores into probabilities | Output layer |
| CBOW | Predict center word from context | Fast, frequent words |
| Skip-gram | Predict context from center word | Rare words, small data |
| Negative sampling | Train vs. a few random negatives | Efficient training |
| Contextual embeddings (BERT) | Word vector varies by context | State-of-the-art NLP |
flowchart TD
subgraph LSI[LSI Pipeline]
A[Term-Document Matrix M
TF-IDF] --> B[SVD: M = K·S·Dᵀ]
B --> C[Keep top-s singular values
reduced concept space]
C --> D[Embed query as pseudo-doc]
D --> E[Cosine rank in concept space]
end
subgraph W2V[Word2Vec Pipeline]
F[Large text corpus] --> G[Generate context-target pairs
self-supervision]
G --> H{Architecture}
H -->|CBOW| I[Predict center from context]
H -->|Skip-gram| J[Predict context from center]
I --> K[Softmax / Negative Sampling]
J --> K
K --> L[Backprop updates WI, WO]
L --> M[Dense word vectors ~300-dim]
end