Custom training loops
Compute parameter derivatives of a loss during model training.
TensorFlow · Class
tf.GradientTape
Records operations performed in a context so TensorFlow can compute gradients automatically. It watches trainable variables by default and supports manual watching, persistent tapes, and higher-order derivatives.
How it works
Use the tape as a context manager around the operations whose derivatives are needed. TensorFlow records eligible operations, then gradient() applies reverse-mode differentiation to a target with respect to selected sources; trainable variables are watched automatically unless that behavior is disabled.
Syntax
tf.GradientTape(persistent=False, watch_accessed_variables=True)Parameters
| Name | Type | Description |
|---|---|---|
persistent | bool | Whether to retain the tape for more than one gradient or Jacobian computation; defaults to False. |
watch_accessed_variables | bool | Whether to automatically watch trainable variables accessed while the tape is active; defaults to True. |
Returns
A GradientTape context-manager object; its gradient(), jacobian(), and related methods return derivatives for the requested targets and sources.
Key points
- GradientTape is a context-managed class for recording computations so TensorFlow can later calculate derivatives.
- Trainable variables are automatically watched when accessed during the tape context.
- Use watch() to include a tensor that is not automatically tracked.
- Nested tapes support derivatives of derivatives, enabling higher-order differentiation.
- Call gradient(target, sources) after the forward pass to obtain derivatives for the chosen sources.
- The constructor defaults to a non-persistent tape with automatic watching enabled.
Common mistakes
- A non-persistent tape is single-use: calling gradient() consumes the tape for gradient computation.
- With automatic watching disabled, explicitly call watch() for every variable whose gradient you need.
- Operations outside the tape context are not recorded for differentiation.
Example: Compute a derivative
import tensorflow as tf
x = tf.Variable(3.0)
with tf.GradientTape() as tape:
y = x * x
dy_dx = tape.gradient(y, x)
print(dy_dx.numpy()) # 6.0Records the forward computation and obtains the derivative of y with respect to x.
When to use it
Input gradients
Differentiate a tensor that is not a trainable variable.
Higher-order differentiation
Calculate second-order derivatives or other nested derivatives.
Interview questions
QHow does tf.GradientTape determine what to differentiate?
GradientTape records differentiable operations executed in its context and uses those records to compute a target's gradient with respect to sources. Trainable variables accessed while the tape is active are watched automatically by default.
Follow-up: How would you differentiate with respect to a plain tensor? Call tape.watch(tensor) before using it in the recorded computation.
QWhen should persistent=True be used?
By default, a tape is released after gradient() is called, so it supports one gradient computation. Construct it with persistent=True when multiple gradient or Jacobian computations must reuse the same recorded work.
Follow-up: When is the persistent tape released? Its resources remain until the tape object is garbage-collected.
QHow can variable watching be controlled?
Set watch_accessed_variables=False to disable automatic tracking of trainable variables. Then explicitly watch each required variable while the tape is active; otherwise the requested gradient cannot be formed from that source.
Follow-up: What happens for disconnected sources? The gradient() method uses unconnected_gradients to control the result, with tf.UnconnectedGradients.NONE as the default.
Related
tf.autodiff.GradientTapetf.GradientTape.gradienttf.GradientTape.jacobiantf.GradientTape.watchtf.UnconnectedGradients