Compact integer indexing
Preparing integer IDs for an embedding layer
Keras · Class
IntegerLookup maps arbitrary integer tokens to contiguous vocabulary indices or categorical encodings. It can use a supplied vocabulary or learn one with adapt().
IntegerLookup builds or accepts a vocabulary of integer tokens, then converts inputs through a table lookup. With the default integer output, known tokens receive contiguous vocabulary indices and unknown tokens use OOV indices; other output modes encode the inputs as categorical vectors or counts.
keras.layers.IntegerLookup(max_tokens=None, num_oov_indices=1, mask_token=None, oov_token=-1, vocabulary=None, vocabulary_dtype="int64", idf_weights=None, invert=False, output_mode="int", sparse=False, pad_to_max_tokens=False, oov_method="floormod", salt=None, name=None, **kwargs)| Name | Type | Description |
|---|---|---|
max_tokens | int or None | Maximum vocabulary size; relevant when adapting or padding to max_tokens. |
num_oov_indices | int | Number of OOV buckets. |
mask_token | int or None | Optional integer used for masking. |
oov_token | int | Fallback token used for OOV values when inverting. |
vocabulary | array-like, tensor, or string | Vocabulary terms, supplied as integer data or a text-file path. |
vocabulary_dtype | string | Integer dtype of vocabulary terms. |
idf_weights | array-like or None | Optional inverse-document-frequency weights for tf_idf output. |
invert | bool | Whether to reverse indices into vocabulary tokens. |
output_mode | string | Output representation: int, one_hot, multi_hot, count, or tf_idf. |
sparse | bool | Whether to return sparse output. |
pad_to_max_tokens | bool | Whether to pad output to max_tokens. |
oov_method | string | OOV hashing method. |
A dense or sparse tensor containing integer indices or the selected categorical encoding.
import keras
import numpy as np
lookup = keras.layers.IntegerLookup(vocabulary=[10, 20, 30])
print(lookup(np.array([20, 99, 10])).numpy())Creates a lookup table from integer vocabulary terms and maps known and unknown values to integer indices.
Preparing integer IDs for an embedding layer
Converting integer features into model-ready categorical vectors
Reversing encoded indices to vocabulary values
IntegerLookup maps arbitrary integer tokens to contiguous indices using a table-backed vocabulary. The vocabulary can be supplied during construction or learned by calling adapt(), which analyzes token frequencies.
Follow-up: How are unseen values handled? They are assigned to one of the configured OOV indices; num_oov_indices controls how many such buckets exist.
The default output_mode is int, so each input becomes its vocabulary index. multi_hot, count, and tf_idf instead produce vectors whose dimensions represent vocabulary entries.
Follow-up: What additional output mode is documented? one_hot is also accepted, and sparse=True can request sparse output where supported.
mask_token can reserve a masking value, while oov_token specifies the token returned when invert=True maps OOV indices back to integer tokens. The default oov_token is -1.
Follow-up: When would you use invert=True? Use it to reverse index lookup and recover vocabulary tokens, while OOV indices map to the configured fallback token.
keras.layers.StringLookupkeras.layers.CategoryEncodingkeras.layers.Embedding