TensorFlow · Class

layers.TextVectorization

A Keras preprocessing layer that turns batches of strings into token IDs or dense text vectors. It can learn its vocabulary from data with adapt() or use one supplied at construction.

How it works

TextVectorization standardizes and splits strings, builds or uses a vocabulary, maps tokens to integer IDs, and emits integer sequences or other vector representations.

Syntax

tf.keras.layers.TextVectorization(max_tokens=None, standardize='lower_and_strip_punctuation', split='whitespace', ngrams=None, output_mode='int', output_sequence_length=None, pad_to_max_tokens=False, vocabulary=None, idf_weights=None, sparse=False, ragged=False, encoding='utf-8', name=None, **kwargs)

Parameters

Name	Type	Description
`max_tokens`	`int or None`	Maximum vocabulary size; limits the number of tokens retained.
`standardize`	`string, callable, or None`	Text normalization callable or supported standardization option.
`split`	`string, callable, or None`	Token-splitting callable or supported split option.
`ngrams`	`int, tuple, or None`	Optional integer specifying n-gram sizes to generate.
`output_mode`	`string`	Output representation mode, such as 'int', 'binary', 'count', 'tf-idf', or 'float'.
`output_sequence_length`	`int or None`	Length used to pad or truncate integer output sequences.
`pad_to_max_tokens`	`bool`	Whether to pad output to max_tokens in applicable dense modes.
`vocabulary`	`array-like or None`	Vocabulary values supplied directly instead of learned with adapt().
`idf_weights`	`array-like or None`	Weights used with TF-IDF output.
`sparse`	`bool`	Whether to return a sparse tensor when supported.
`ragged`	`bool`	Whether to return ragged tensors for variable-length output.
`encoding`	`string`	Input string encoding.

Returns

A tensor representation of the input strings: integer token sequences or dense floating-point vectors, with sparse or ragged output available when configured.

Key points

The class is a Keras preprocessing layer for converting text into numerical representations.
Inputs are batches of strings, with each individual example represented by one string.
adapt() scans training text, counts string values, and constructs the vocabulary.

Common mistakes

The layer cannot vectorize meaningfully until its vocabulary is supplied or learned with adapt().

Example: Learn a vocabulary and encode text

import tensorflow as tf

layer = tf.keras.layers.TextVectorization(
    max_tokens=1000,
    output_mode="int",
    output_sequence_length=8,
)
layer.adapt(tf.data.Dataset.from_tensor_slices([
    "TensorFlow makes text useful",
    "Text becomes integer sequences",
]).batch(2))

encoded = layer(tf.constant(["Text is useful"]))
print(encoded.shape)
print(layer.get_vocabulary()[:5])

Learns a vocabulary from text, then converts new strings into fixed-length integer sequences.

When to use it

Integer token sequences

Prepare raw text for an Embedding layer.

Dense text features

Convert documents into fixed-dimensional features for a dense model.

Binary, count, or TF-IDF features

Represent token presence or weighting for classical text models.

Interview questions

QWhat does TextVectorization do, and how is its vocabulary obtained?

It processes each example by standardizing it, splitting it, optionally recombining pieces into n-grams, indexing tokens, and producing either integer or dense representations. The vocabulary must be provided or learned with adapt().

Follow-up: How would you keep sequence shapes fixed? Set output_sequence_length; the layer pads or truncates integer sequences to that length.

QHow do output modes differ?

output_mode='int' returns token IDs and preserves token order. Other supported modes produce dense representations such as binary, count, frequency, or TF-IDF-style outputs.

Follow-up: When is int preferable? Use it when an Embedding layer should consume ordered token IDs.

QWhat are the default text-processing steps?

The default standardization lowercases text and strips punctuation, while the default split behavior uses whitespace. Custom standardization and splitting functions can change that behavior.

Follow-up: What should a custom standardizer return? It should return transformed strings with the same batch structure expected by the layer.

tf.keras.layers.StringLookup
tf.keras.layers.CategoryEncoding
tf.keras.layers.Embedding
tf.keras.layers.TextVectorization.adapt
tf.keras.layers.TextVectorization.get_vocabulary

Source: tf.keras.layers.TextVectorization | TensorFlow v2.16.1. Checked 2026-10-15.