Prevent gradient accumulation
Before processing each training batch
PyTorch · Module
optimizer.zero_grad
Resets gradients for all tensors optimized by an optimizer. By default, it sets gradients to None instead of zero, usually reducing memory use and sometimes improving performance.
How it works
The method clears gradients for every tensor managed by the optimizer. By default it replaces each gradient with None rather than a zero tensor, which can reduce memory use and improve performance; passing False keeps zero tensors instead.
Syntax
Optimizer.zero_grad(*set_to_none=True*)Parameters
| Name | Type | Description |
|---|---|---|
set_to_none | bool, optional | If True, assign None to gradients instead of setting their values to zero. Defaults to True. |
Returns
The documentation does not specify a return value; it performs the gradient reset in place.
Key points
- It clears gradients belonging to every tensor registered with the optimizer.
- set_to_none=True replaces gradients with None rather than writing zero values into existing gradient tensors.
- The default is True and commonly uses less memory.
- Using None can provide a modest performance benefit.
- After clearing with None and running backward, parameters that receive no gradient retain grad=None.
Common mistakes
- With the default, a parameter's grad may be None, so code that directly reads or modifies gradients must handle None rather than assuming a zero-filled tensor.
- A zero gradient still permits an optimizer step, whereas a None gradient can cause the optimizer to skip that parameter's update.
Example: Clear gradients in a training step
import torch
from torch import nn
model = nn.Linear(2, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
x = torch.tensor([[1.0, 2.0]])
y = torch.tensor([[1.0]])
optimizer.zero_grad()
loss = ((model(x) - y) ** 2).mean()
loss.backward()
optimizer.step()The optimizer clears gradients before each backward pass, then uses the newly computed gradients to update the model.
When to use it
Use None gradients
When optimizing memory usage
Choose explicit zero tensors
When manually inspecting or editing gradients
Interview questions
QWhat does optimizer.zero_grad() do, and where does it appear in a training loop?
It resets gradients for all optimized tensors. It is normally called before computing the next batch's backward pass because gradients otherwise accumulate.
Follow-up: What does set_to_none=True change? It stores None instead of zero tensors, generally reducing memory use and potentially improving performance, but callers must handle missing gradients.
QWhat is the purpose and default of set_to_none?
set_to_none controls whether gradients are assigned None or filled with zeros. Its default is True.
Follow-up: Why can None matter to an optimizer? Optimizers may skip a parameter when its gradient is None, while a zero gradient can still produce a step.
QHow does zero_grad relate to backward() and step()?
The method affects all parameters included in the optimizer. It does not compute gradients; backward() computes them, and step() applies them.
Follow-up: How would you clear gradients while preserving zero-valued gradient tensors? Call zero_grad(set_to_none=False).
Related
torch.optim.Optimizeroptimizer.step()Tensor.backward()torch.optim.Optimizer.add_param_group