Bag-of-words features
Turn labeled or unlabeled text documents into numeric inputs for a classifier or other estimator.
Scikit-Learn · Class
CountVectorizer
CountVectorizer converts text documents into a sparse matrix of token-occurrence features. It learns a vocabulary unless one is supplied and supports configurable preprocessing, tokenization, stop-word filtering, and n-grams.
How it works
CountVectorizer learns a token-to-column vocabulary from documents and converts each document into a sparse matrix whose entries are token occurrence counts. Its configurable analysis pipeline can decode, preprocess, tokenize, filter stop words, create word or character n-grams, and limit vocabulary features.
Syntax
class sklearn.feature_extraction.text.CountVectorizer(*, input='content', encoding='utf-8', decode_error='strict', strip_accents=None, lowercase=True, preprocessor=None, tokenizer=None, stop_words=None, token_pattern='(?u)\\b\\w\\w+\\b', ngram_range=(1, 1), analyzer='word', max_df=1.0, min_df=1, max_features=None, vocabulary=None, binary=False, dtype=<class 'numpy.int64'>)Parameters
| Name | Type | Description |
|---|---|---|
input | {'filename', 'file', 'content'}, default='content' | Input documents: filenames, file objects, or content strings, depending on input. |
encoding | str, default='utf-8' | Character encoding used to decode input files. |
decode_error | {'strict', 'ignore', 'replace'}, default='strict' | How decoding errors are handled. |
strip_accents | {'ascii', 'unicode'}, callable, default=None | Accent-stripping mode, or None. |
lowercase | bool, default=True | Convert characters to lowercase before tokenization. |
preprocessor | callable, default=None | Callable preprocessing function, or None. |
tokenizer | callable, default=None | Callable tokenization function, or None. |
stop_words | {'english'}, list, default=None | Stop-word list, the string 'english', or None. |
token_pattern | str, default=r'(?u)\b\w\w+\b' | Regular expression selecting tokens when analyzer='word'. |
ngram_range | tuple (min_n, max_n), default=(1, 1) | Inclusive minimum and maximum n-gram sizes. |
analyzer | {'word', 'char', 'char_wb'} or callable, default='word' | Feature analyzer: word, character, character-within-word, or callable. |
max_df | float or int, default=1.0 | Ignore terms above this document-frequency threshold. |
Returns
fit_transform and transform return a sparse document-term matrix of shape (n_samples, n_features), with CSR-array storage and the configured dtype.
Key points
- The class turns a document collection into numeric count features suitable for downstream estimators.
- The transformed result is sparse and uses CSR-array storage in the current stable documentation.
- Omitting vocabulary causes the estimator to learn token indices from the supplied documents.
- ngram_range controls the smallest and largest n-gram sizes included in the feature matrix.
- max_df removes terms appearing in too many documents, using either a proportion or an absolute document count.
Common mistakes
- A custom analyzer replaces the normal preprocessing and tokenization path, so configure the callable with the intended input handling.
Example: Count words and bigrams
from sklearn.feature_extraction.text import CountVectorizer
corpus = ["red fish fish", "blue fish"]
vectorizer = CountVectorizer(ngram_range=(1, 2), binary=False)
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names_out().tolist())
print(X.toarray().tolist())Fits a vocabulary on two documents and transforms them into sparse unigram-and-bigram count vectors.
When to use it
Word n-grams
Represent short phrases and local word context.
Character features
Capture spelling fragments, prefixes, or character-level patterns.
Binary occurrence features
Replace raw frequencies with document-level presence indicators.
Interview questions
QWhat does CountVectorizer produce?
CountVectorizer converts text documents into a matrix of token counts and stores the learned feature-to-column mapping in its vocabulary. The matrix uses a sparse representation, which avoids allocating values for absent tokens.
Follow-up: How would you represent presence instead of frequency? Set binary=True; nonzero counts are then replaced by 1 while the vocabulary and feature analysis remain available.
QHow do fit, transform, and fit_transform differ?
fit learns the vocabulary and analyzer state from the training documents. fit_transform learns that state and returns the transformed training matrix in one operation; transform applies the learned mapping to new documents.
Follow-up: Why must a test set use transform rather than fit_transform? Refitting would construct a different vocabulary and can leak information from the test corpus into feature construction.
QHow do ngram_range and analyzer affect features?
ngram_range specifies the lower and upper boundary of the n-gram sizes generated by the analyzer. For example, (1, 2) includes unigrams and bigrams, while analyzer determines whether word or character features are extracted.
Follow-up: How can vocabulary growth be controlled? Use max_features to keep only the most frequent features, or min_df and max_df to filter features by document frequency.
Related
TfidfVectorizerTfidfTransformerHashingVectorizerTfidfTransformer