Source: 07_Text_class.pdf
Text classification assigns documents to predefined classes (labeled groups), formally a binary function F: D × C → {0,1}. When no labels exist and documents are just partitioned by similarity, it’s clustering (unsupervised). The approach depends on machine learning type: supervised (learn from human-labeled training data), unsupervised (no labels), or semi-supervised (small labels + much unlabeled data). A classifier is trained on a training set and evaluated on an unseen test set; too much fitting to training data causes overfitting.
Unsupervised methods include K-means (assign docs to nearest of K centroids, recompute, repeat), bisecting K-means, and hierarchical clustering (merge closest clusters using single/complete/average link). Supervised methods are richer: Decision Trees (split on high information-gain terms), kNN (lazy — classify by the majority class of nearest neighbors), the Rocchio classifier (build a class centroid from positive/negative feedback), Naive Bayes (probabilistic, assumes term independence; multinomial variant uses term frequencies), and Support Vector Machines (find the max-margin separating hyperplane defined by support vectors; use soft margins or kernels when not linearly separable). Ensembles combine classifiers via stacking or boosting.
Evaluation uses a contingency table. Plain accuracy/error is misleading on skewed classes (predicting “none in class” can score 98%), so precision and recall per class, combined as the F-measure (F1), are preferred — averaged via micro (per-document) or macro (per-class). Cross-validation (commonly 10-fold) gives statistically valid results. Standard benchmark collections include Reuters-21578/RCV1, OHSUMED, and 20 Newsgroups. Classes can be organized into taxonomies (hierarchical, expert-built) or folksonomies (collaborative user tags).
| Term | Meaning | Use Case |
|---|---|---|
| Text classification | Assign docs to predefined classes | Spam filtering, topic labeling |
| Clustering | Group docs without labels | Auto-organizing results |
| Supervised / unsupervised | Learn from labels / no labels | Choice of method |
| Overfitting | Classifier too specific to training data | Caution; use test set |
| K-means | Partition into K centroid-based clusters | Document grouping |
| Hierarchical clustering | Merge closest clusters into a tree | Cluster taxonomies |
| Decision Tree | Tree of term-based splitting rules | Interpretable classification |
| kNN | Classify by k nearest neighbors (lazy) | On-demand classification |
| Rocchio classifier | Class centroid from positive/negative terms | Centroid-based classification |
| Naive Bayes | Probabilistic, term-independence assumption | Fast baseline classifier |
| SVM | Max-margin separating hyperplane | High-accuracy text classification |
| Support vectors | Docs defining the delimiting hyperplanes | Define SVM boundary |
| Kernel / soft margin | Handle non-linearly-separable data | Complex boundaries |
| Stacking / boosting | Ensemble combination methods | Higher precision |
| F1 (micro/macro) | Combined precision/recall, averaged | Classifier evaluation |
| Cross-validation | k-fold train/test rotation | Statistical validation |
| Taxonomy / folksonomy | Expert hierarchy / collaborative tags | Organizing classes |
flowchart TD
A[Document Collection] --> B{Labels available?}
B -- No --> C[Unsupervised: Clustering
K-means / hierarchical]
B -- Yes --> D[Training Set
human-labeled]
D --> E[Train Classifier
DT / kNN / NB / SVM]
E --> F[Validate on Test Set]
F --> G{Good metrics?}
G -- No --> H[Tune / cross-validate]
H --> E
G -- Yes --> I[Classify new documents]
C --> J[Assign labels to clusters]
F --> K[Evaluate: Precision/Recall/F1
micro & macro]