PyTorch · Class

optim.Adam

torch.optim.Adam implements the Adam optimization algorithm for neural-network parameters. It supports configurable learning rate, moment coefficients, numerical stability, weight decay, AMSGrad, and multiple execution modes.

How it works

Adam maintains running estimates derived from each parameter's gradient and squared gradient, then uses bias-corrected estimates to update parameters. Its options control the learning rate, smoothing coefficients, numerical-stability term, regularization, AMSGrad, execution strategy, maximization, graph capture, differentiability, fused execution, and decoupled weight decay.

Syntax

class torch.optim.Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False, *, foreach=None, maximize=False, capturable=False, differentiable=False, fused=None, decoupled_weight_decay=False)

Parameters

Name	Type	Description
`params`	`iterable`	Iterable of parameters or named parameters to optimize, or dictionaries defining parameter groups.
`lr`	`float or Tensor, optional`	Learning rate; default is 1e-3.
`betas`	`tuple[float or Tensor, float or Tensor], optional`	Coefficients for running averages of the gradient and its square; default is (0.9, 0.999).
`eps`	`float, optional`	Denominator term for numerical stability; default is 1e-8.
`weight_decay`	`float, optional`	Weight-decay coefficient; default is 0.
`amsgrad`	`bool, optional`	Whether to use AMSGrad; default is False.
`foreach`	`bool, optional`	Whether to use the foreach implementation; default is None.
`maximize`	`bool, optional`	Whether to maximize the objective instead of minimize it; default is False.
`capturable`	`bool, optional`	Whether optimizer execution must support CUDA graph capture; default is False.
`differentiable`	`bool, optional`	Whether to construct a differentiable optimizer step; default is False.
`fused`	`bool, optional`	Whether to use the fused implementation; default is None.
`decoupled_weight_decay`	`bool, optional`	Whether to use decoupled weight decay; default is False.

Key points

Adam is available as a standard optimizer class for updating trainable parameters.
The input can be parameters, named parameters, or dictionaries that define separate parameter groups.
The default learning rate is 0.001; provide lr to change the global step size.
betas controls smoothing for the first- and second-moment estimates.
eps stabilizes division in the parameter update.
Set amsgrad to enable the AMSGrad variant.

Common mistakes

A tensor learning rate does not work with every Adam implementation; use a scalar learning rate unless the selected implementation supports it.
The documented supported floating-point parameter types are float64, float32, float16, and bfloat16.
If parameter groups use named parameters, every parameter across all groups must have a name.

Example: Train one step with Adam

import torch
from torch import nn

model = nn.Linear(4, 1)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

x = torch.randn(8, 4)
y = torch.randn(8, 1)
loss = nn.functional.mse_loss(model(x), y)
loss.backward()
optimizer.step()
optimizer.zero_grad()

Creates an Adam optimizer for a linear model, computes a loss, backpropagates gradients, updates parameters, and clears gradients.

When to use it

Default model optimization

Training neural networks with a general-purpose adaptive optimizer.

Parameter groups

Optimizing models whose parameters need separate hyperparameters.

Specialized execution

Using CUDA graph capture or differentiable optimization workflows.

Interview questions

QWhat does torch.optim.Adam optimize, and what is its default learning rate?

Adam is an adaptive optimizer that uses running averages of gradients and squared gradients to scale parameter updates. Its constructor accepts parameter iterables or parameter-group dictionaries, and the default learning rate is 0.001.

Follow-up: How would you apply different learning rates? Pass an iterable of parameter-group dictionaries, each containing its own params and optimizer options such as lr.

QWhat are betas and eps used for?

betas contains the coefficients for the running averages of the gradient and its square. The defaults are (0.9, 0.999), while eps adds a small denominator term for numerical stability.

Follow-up: Why might eps matter? It prevents division by a value that is zero or too small during an update.

QWhich constructor options alter Adam's optimization behavior?

amsgrad selects the AMSGrad variant, while decoupled_weight_decay separates weight decay from the adaptive gradient update. maximize reverses the optimization direction by maximizing the objective instead of minimizing it.

Follow-up: When is decoupled weight decay useful? It provides AdamW-style regularization rather than adding L2 penalty directly to the gradient.

torch.optim.Optimizer
torch.optim.AdamW
torch.optim.Adamax
torch.optim.SparseAdam
torch.optim.adam.adam

Source: Adam — PyTorch main documentation. Checked 2026-10-16.