Input pipelines
Preparing batched inputs for model training or evaluation.
TensorFlow · Class
Dataset
tf.data.Dataset represents a collection of elements and a pipeline of transformations. Build one from input data, transform it with operations such as map or batch, and iterate over the resulting elements.
How it works
A Dataset represents elements together with a logical pipeline of transformations. Create it from tensors or another source, apply operations such as map and batch, then iterate over the resulting elements for model input or other processing.
Syntax
@staticmethod
from_tensor_slices(tensors, name=None) -> 'DatasetV2'Parameters
| Name | Type | Description |
|---|---|---|
tensors | tf.Tensor or nested structure | Input tensors or compatible nested tensor structure to slice along the first dimension. |
name | str or None | Optional operation name. |
Returns
A DatasetV2 whose elements are slices of tensors along their first dimension.
Key points
- A normal workflow creates a source dataset, applies preprocessing transformations, and iterates over the processed elements.
- from_tensor_slices constructs elements by slicing supplied tensor-like inputs.
- Slicing occurs on axis zero, so each leading-dimension slice becomes a dataset element.
- Nested structures such as tuples or dictionaries remain structurally intact, with only their leading dimension removed.
- The documented factory is a static method accepting tensors and an optional name, returning DatasetV2.
Common mistakes
- When using from_tensor_slices with multiple tensors, their leading dimensions must have equal lengths.
- Choose flat_map when flattening nested datasets while preserving the source order matters.
Example: Build and transform a dataset
import tensorflow as tf
dataset = tf.data.Dataset.from_tensor_slices(([1, 2, 3], [10, 20, 30]))
dataset = dataset.map(lambda x, y: (x * 2, y)).batch(2)
for features, labels in dataset.as_numpy_iterator():
print(features, labels)Creates paired elements, transforms the feature values, batches them, and iterates as NumPy arrays.
When to use it
Preprocessing
Applying feature conversion or normalization before consumption.
Dataset composition
Flattening nested datasets into a sequential stream.
Interview questions
QWhat is tf.data.Dataset used for?
Dataset is an abstraction for a collection of elements plus a logical plan of transformations. A typical pipeline creates a source, transforms it for preprocessing, and iterates over the result.
Follow-up: How would you flatten datasets while retaining order? Use flat_map, which the documentation specifically recommends for that requirement.
QHow does from_tensor_slices work?
from_tensor_slices creates one dataset element per slice along the first dimension. It preserves the input structure while removing that leading dimension from each element.
Follow-up: What constraint applies when slicing multiple tensors? Their first dimensions must have identical sizes.
QWhat operations commonly follow dataset creation?
A practical pipeline can map preprocessing over elements and then batch them before iteration. The resulting dataset can be consumed with an iterator or as_numpy_iterator for NumPy-compatible output.
Follow-up: How can preprocessing be parallelized? Pass num_parallel_calls to map, commonly using a suitable parallelism value such as tf.data.AUTOTUNE.
Related
tf.data.Dataset.maptf.data.Dataset.batchtf.data.Dataset.flat_maptf.data.Dataset.as_numpy_iteratortf.data.Dataset.get_single_element