Keras · Class

layers.TextVectorization

TextVectorization turns batches of strings into integer token sequences or dense text-feature vectors. It standardizes, splits, optionally creates n-grams, indexes tokens, and applies the selected output mode.

How it works

TextVectorization processes each string by standardizing it, splitting it, optionally forming n-grams, assigning vocabulary indices, and producing either integer sequences or dense numeric representations. Its vocabulary is supplied at construction or learned with adapt().

Syntax

keras.layers.TextVectorization(max_tokens=None, standardize="lower_and_strip_punctuation", split="whitespace", ngrams=None, output_mode="int", output_sequence_length=None, pad_to_max_tokens=False, vocabulary=None, idf_weights=None, sparse=False, ragged=False, encoding="utf-8", name=None, **kwargs)

Parameters

Name	Type	Description
`max_tokens`	`int or None`	Maximum vocabulary size.
`standardize`	`str, callable, or None`	Standardization operation or callable.
`split`	`str, callable, or None`	Token splitting operation or callable.
`ngrams`	`int, tuple, or None`	Optional n-gram sizes.
`output_mode`	`str`	Output representation mode: int, multi_hot, count, or tf_idf.
`output_sequence_length`	`int or None`	Length used for integer output sequences.
`pad_to_max_tokens`	`bool`	Whether to pad the vocabulary-sized output.
`vocabulary`	`array-like or None`	Vocabulary supplied directly instead of learned.
`idf_weights`	`array-like or None`	IDF weights used with TF-IDF output.
`sparse`	`bool`	Whether output is sparse.
`ragged`	`bool`	Whether output is ragged.
`encoding`	`str`	Text encoding.

Returns

A tensor containing integer token indices or a dense numeric representation, depending on output_mode.

Key points

Use the layer to turn raw text features into numerical representations suitable for subsequent neural-network layers.
Inputs are batches of strings, with each individual example represented by one string.
The first stage normally lowercases text and removes punctuation, although standardization can be configured.
After splitting, the layer can combine substrings into n-gram tokens.
Select integer sequences, multi-hot vectors, count vectors, or TF-IDF vectors with output_mode.

Common mistakes

Compiled models using a backend other than TensorFlow cannot include this layer in their computation graph; eager execution remains supported.
The layer is not ready to produce meaningful vocabulary-based output until its vocabulary is provided or learned.

Example: Adapt and vectorize text

import keras
from keras import layers

vectorizer = layers.TextVectorization(
    max_tokens=1000,
    output_mode="int",
    output_sequence_length=20,
)
vectorizer.adapt(["Keras makes text processing simple.", "Text becomes integers."])
print(vectorizer(["Keras text processing"]).numpy())

Creates a vocabulary from sample text, then converts new strings into fixed-length integer token sequences.

When to use it

Token-index sequences

Preparing raw text for a neural network

Count or multi-hot features

Representing documents as fixed-size numeric features

TF-IDF vectors

Building classical text classification features

Integrated inference preprocessing

Keeping preprocessing with a TensorFlow model

Interview questions

QWhat processing pipeline does TextVectorization apply?

It converts batches of strings into token-index sequences or dense token statistics. Processing consists of standardization, splitting, optional n-gram recombination, indexing, and output transformation.

Follow-up: How would you create the vocabulary from training data? Call adapt() on the training text, then reuse the same layer for later data so the learned mapping stays fixed.

QWhat output modes are available?

output_mode="int" produces integer token indices. The other documented modes are multi_hot, count, and tf_idf, which produce dense numeric representations of token presence, frequency, or TF-IDF values.

Follow-up: When would tf_idf be preferable to integer output? Use it when the downstream model should receive a fixed-size weighted feature vector rather than a token sequence.

QWhat backend limitation should you know?

The layer can run eagerly with any Keras backend, but it uses TensorFlow internally and cannot be part of a compiled computation graph for non-TensorFlow backends.

Follow-up: What deployment constraint follows from that limitation? Keep this preprocessing outside a compiled non-TensorFlow graph, or use it in eager execution or a TensorFlow-backed compiled model.

StringLookup layer
Embedding layer
Preprocessing layers

Source: TextVectorization layer - Keras. Checked 2026-10-15.