Text classification features
Representing documents for supervised learning
Scikit-Learn · Class
TfidfVectorizer
TfidfVectorizer converts raw documents into a sparse matrix of TF-IDF features. It combines CountVectorizer and TfidfTransformer functionality in one estimator.
How it works
The vectorizer tokenizes raw documents, builds a vocabulary, counts token occurrences, applies inverse-document-frequency weighting, and normalizes the resulting sparse TF-IDF vectors. It combines CountVectorizer and TfidfTransformer behavior in one estimator.
Syntax
TfidfVectorizer(*, input='content', encoding='utf-8', decode_error='strict', strip_accents=None, lowercase=True, preprocessor=None, tokenizer=None, analyzer='word', stop_words=None, token_pattern='(?u)\\b\\w\\w+\\b', ngram_range=(1, 1), max_df=1.0, min_df=1, max_features=None, vocabulary=None, binary=False, dtype=<class 'numpy.float64'>, norm='l2', use_idf=True, smooth_idf=True, sublinear_tf=False)Parameters
| Name | Type | Description |
|---|---|---|
raw_documents | iterable | Input documents; each item is a string or, with a callable analyzer, another supported input representation. |
max_df | float or int, default=1.0 | Maximum document-frequency threshold for retaining terms, as a proportion or count. |
min_df | float or int, default=1 | Minimum document-frequency threshold for retaining terms, as a proportion or count. |
max_features | int, default=None | Maximum number of features to keep when learning the vocabulary. |
use_idf | bool, default=True | Whether to compute inverse-document-frequency weights. |
norm | {'l1', 'l2'} or None, default='l2' | Normalization applied to output vectors; commonly 'l1', 'l2', or None. |
smooth_idf | bool, default=True | Whether to smooth the IDF calculation. |
sublinear_tf | bool, default=False | Whether to replace raw term frequency with 1 plus its logarithm. |
Returns
A sparse matrix of shape (n_samples, n_features) containing TF-IDF features.
Key points
- It accepts raw text documents and returns a document-by-feature TF-IDF representation.
- The class packages count-based text extraction and TF-IDF transformation into one estimator.
- use_idf controls whether terms receive corpus-level inverse-document-frequency weighting.
Common mistakes
- max_df removes terms appearing in too many documents; its meaning depends on whether it is a float proportion or integer count.
- Fixed vocabularies prevent max_features from limiting the learned feature set.
Example: Fit a TF-IDF feature matrix
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = ["cats chase mice", "dogs chase cats"]
vectorizer = TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True)
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names_out())
print(X.shape)Fits the vocabulary and TF-IDF weighting on two documents, including unigrams and bigrams, then prints the learned feature names and sparse matrix shape.
When to use it
Document clustering
Grouping documents by their vocabulary and weighting
Similarity and retrieval
Comparing document vectors after normalization
Interview questions
QWhat does TfidfVectorizer do?
It converts raw documents into a TF-IDF feature matrix and is equivalent to applying CountVectorizer followed by TfidfTransformer. The result is normally a sparse matrix.
Follow-up: How would you reproduce it explicitly? Use CountVectorizer to create counts, then pass those counts to TfidfTransformer with matching settings.
QWhat is the effect of use_idf?
use_idf enables inverse-document-frequency reweighting. With it disabled, the IDF value for every term is 1, so only term frequency and normalization affect the representation.
Follow-up: What does smooth_idf change? It adds one to document counts in the IDF formula, preventing zero division and altering the weighting for rare terms.
QWhat is the difference between fit, transform, and fit_transform?
fit learns the vocabulary and corpus-dependent IDF values, while transform applies those learned values to new documents. fit_transform performs both operations on the supplied corpus.
Follow-up: Why must fitting be restricted to training data? Fitting on evaluation documents leaks vocabulary and corpus statistics into the evaluation procedure.
Related
CountVectorizerTfidfTransformerHashingVectorizer