Build a source pipeline
Prepare tensors or in-memory values for model input.
TensorFlow · Class
tf.data.Dataset
tf.data.Dataset represents a potentially large sequence of elements and provides composable operations for building TensorFlow input pipelines. Data is commonly sourced, transformed, and then consumed by iteration.
How it works
A Dataset represents a sequence of elements that can be produced from tensors or other sources and processed through chained transformations. Iteration consumes elements progressively, allowing pipelines to handle data without loading the entire collection into memory.
Syntax
tf.data.Dataset(variant_tensor)Parameters
| Name | Type | Description |
|---|---|---|
variant_tensor | tf.Tensor | The variant tensor backing the dataset object. |
Returns
A tf.data.Dataset representing a potentially large set of elements.
Key points
- Dataset models a sequence that may contain many elements and can serve as the foundation of an input pipeline.
- Its API is designed to make input pipelines both readable and efficient.
- A pipeline begins by constructing a source from available input data.
- Transformations provide the stage where records can be prepared before model or application code consumes them.
- Consumers process the pipeline by iterating over its produced elements.
- For small in-memory examples, a Python list can be converted into a Dataset with a source factory.
Common mistakes
- Use from_tensor_slices rather than from_tensors when each item in an input tensor should become a separate dataset element.
- Streaming avoids requiring the complete dataset in memory, but iteration still consumes the pipeline progressively rather than materializing all results at once.
Example: Transform and batch a dataset
import tensorflow as tf
ds = tf.data.Dataset.from_tensor_slices([1, 2, 3])
ds = ds.map(lambda x: x * 2).batch(2)
for batch in ds:
print(batch.numpy())Creates a dataset, transforms each element, groups elements into batches, and iterates over the resulting pipeline.
When to use it
Transform training examples
Apply preprocessing operations such as mapping and batching before consumption.
Stream large datasets
Process collections too large to load into memory at once.
Supply model data
Feed transformed batches into training or evaluation loops.
Interview questions
QWhat is tf.data.Dataset used for?
A Dataset is an iterable sequence of elements used to build TensorFlow input pipelines. A typical pipeline creates a source, applies transformations, and iterates over the resulting elements.
Follow-up: Why is this useful for large inputs? Iteration is streaming, so the complete dataset does not have to fit in memory.
QDescribe the common Dataset workflow.
The usual workflow is to create a source dataset, transform it for preprocessing, and iterate over it to feed computation. Transformations can be chained to express the input pipeline.
Follow-up: How would you turn individual tensor entries into examples? Use from_tensor_slices rather than from_tensors.
QHow do from_tensors and from_tensor_slices differ?
from_tensors creates a dataset with one element whose structure contains the supplied tensors. from_tensor_slices instead divides the input along its first dimension and creates multiple elements.
Follow-up: Which constructor is appropriate for a list such as [1, 2, 3]? from_tensor_slices produces one dataset element for each value.
Related
tf.data.Dataset.from_tensor_slicestf.data.Dataset.from_tensorstf.data.Dataset.maptf.data.Dataset.batchtf.data.Dataset.shuffletf.data.Dataset.prefetchtf.data.Dataset.take