Standard gradient-based training
Train neural-network parameters with minibatch gradients.
PyTorch · Module
torch.optim
torch.optim provides optimization algorithms for updating model parameters from computed gradients. Create an optimizer with parameters and algorithm-specific options, then use it in the training loop to apply updates.
How it works
torch.optim supplies optimizer objects that retain optimization state and modify model parameters using their computed gradients. Construct one with model parameters or named parameters, configure algorithm-specific options such as the learning rate, then clear gradients, backpropagate a loss, and call step(); learning-rate schedulers can adjust rates across epochs.
Syntax
torch.optim.<Optimizer>(params, **kwargs)Parameters
| Name | Type | Description |
|---|---|---|
params | iterable | Iterable of torch.nn.Parameter objects, named parameters, or dictionaries defining parameter groups. |
optimizer options | keyword arguments | Algorithm-specific settings such as learning rate and weight decay; exact options depend on the selected optimizer. |
Returns
An optimizer object that stores state and updates the supplied parameters.
Key points
- The module contains implementations of multiple optimization algorithms.
- Common optimization methods are available directly rather than requiring custom integration.
- An optimizer object stores the state needed across update steps.
- Construction accepts parameters, named parameters, or parameter-group dictionaries.
- Each optimizer can receive algorithm-specific settings including learning rate and weight decay.
- Learning-rate schedulers provide APIs for changing the learning rate during training.
Common mistakes
- Algorithms such as LBFGS may require a closure because they reevaluate the model and loss multiple times.
- A closure must reset gradients, recompute the loss, and return that loss.
- Calling step() before gradients are computed does not perform the intended gradient-based update.
Example: Basic optimizer training step
import torch
from torch import nn
model = nn.Linear(2, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
x = torch.tensor([[1.0, 2.0]])
y = torch.tensor([[1.0]])
optimizer.zero_grad()
loss = ((model(x) - y) ** 2).mean()
loss.backward()
optimizer.step()Creates an SGD optimizer, computes a loss, backpropagates gradients, and updates the model parameters.
When to use it
Per-parameter-group configuration
Use different learning rates or regularization settings for parameter subsets.
Learning-rate scheduling
Change the learning rate as training progresses.
Closure-based optimization
Optimize objectives requiring repeated model evaluation.
Interview questions
QWhat is torch.optim?
torch.optim is a package of optimization algorithms. An optimizer holds current optimization state and updates parameters using computed gradients.
Follow-up: How can learning rates change during training? Use torch.optim.lr_scheduler.LRScheduler methods based on the number of epochs.
QHow is an optimizer used in a training loop?
Construct an optimizer with an iterable of Parameter objects or named parameters, then provide algorithm-specific options such as learning rate and weight decay. During training, clear gradients, compute gradients with backpropagation, and call step().
Follow-up: Why can parameter groups be useful? They allow different optimization options for separate sets of parameters.
QWhen is a closure needed?
Most optimizers use the straightforward step() form after gradients are computed. Algorithms that reevaluate the objective, such as LBFGS, instead receive a closure that clears gradients, computes the loss, and returns it.
Follow-up: What must the closure return? It must return the recomputed loss so the optimizer can use the reevaluated objective.
Related
torch.optim.Optimizertorch.optim.lr_scheduler.LRSchedulertorch.optim.SGDtorch.optim.Adamtorch.optim.AdamW