PyTorch · Class

optim.RMSprop

RMSprop is a PyTorch optimizer that scales parameter updates using a moving average of squared gradients. It also supports momentum, weight decay, centered normalization, graph capture, maximization, and differentiable optimization steps.

How it works

RMSprop maintains a moving average of squared gradients and uses it to scale each parameter’s update. Optional momentum, weight decay, and centered variance normalization modify that update; maximize reverses the optimization direction, while differentiable enables autograd through the step.

Syntax

class torch.optim.RMSprop(params, lr=0.01, alpha=0.99, eps=1e-08, weight_decay=0, momentum=0, centered=False, capturable=False, foreach=None, maximize=False, differentiable=False)

Parameters

Name	Type	Description
`params`	`iterable`	Iterable of parameters, named parameters, or parameter-group dictionaries to optimize.
`lr`	`float or Tensor, optional`	Learning rate; default 0.01.
`alpha`	`float, optional`	Smoothing constant; default 0.99.
`eps`	`float, optional`	Denominator stabilizer; default 1e-8.
`weight_decay`	`float, optional`	L2 penalty; default 0.
`momentum`	`float, optional`	Momentum factor; default 0.
`centered`	`bool, optional`	Whether to use centered RMSprop.
`capturable`	`bool, optional`	Whether the optimizer is safe for graph capture, including CUDA graphs or torch.compile.
`foreach`	`bool, optional`	Whether to use the foreach implementation when available.
`maximize`	`bool, optional`	Whether to maximize the objective instead of minimize it.
`differentiable`	`bool, optional`	Whether to record the optimizer step through autograd.

Returns

An RMSprop optimizer object that stores optimizer state and updates the supplied parameters.

Key points

torch.optim.RMSprop is the optimizer class for RMSprop updates.
Each parameter’s effective learning rate is scaled by the inverse square root of its running squared-gradient average, with epsilon for stability.
The default learning rate is 0.01, although a different value is commonly selected for a particular model and task.
Alpha defaults to 0.99 and determines the smoothing used for the squared-gradient moving average.
Epsilon prevents unstable division when the running gradient statistic is very small.
Setting centered=True uses a centered variant that normalizes gradients using an estimate of their variance.

Common mistakes

PyTorch places epsilon after the square root, so implementations that add it before the square root are not equivalent.
If named parameters are supplied across parameter groups, every parameter must have a name.
Optimizer steps are not tracked by autograd unless differentiable=True is used.

Example: One RMSprop update

import torch

model = torch.nn.Linear(10, 1)
optimizer = torch.optim.RMSprop(model.parameters(), lr=1e-3)

x = torch.randn(8, 10)
y = torch.randn(8, 1)
loss = torch.nn.functional.mse_loss(model(x), y)
optimizer.zero_grad()
loss.backward()
optimizer.step()

Creates an RMSprop optimizer for a linear model and performs one standard gradient-update step.

When to use it

Adaptive per-parameter updates

Training models whose gradient scales vary substantially across parameters.

Non-stationary optimization

Optimizing objectives with recurrent or non-stationary gradient behavior.

Graph-captured training

Running optimizer steps inside CUDA graphs or torch.compile workflows.

Differentiable optimization

Learning optimization policies or other procedures requiring gradients through updates.

Interview questions

QHow does RMSprop adapt the learning rate?

RMSprop adapts parameter updates using a weighted moving average of squared gradients. Its effective learning rate is the scheduled learning rate divided by the square root of that average plus epsilon.

Follow-up: How does centered RMSprop differ? It also tracks an average gradient and normalizes using an estimate of gradient variance rather than only the second-moment average.

QWhat are alpha and eps used for?

The smoothing constant alpha controls the moving average of squared gradients, with a default of 0.99. Epsilon is added to the denominator for numerical stability, and its default is 1e-8.

Follow-up: What happens if epsilon is changed? It changes the denominator’s stabilizing offset and therefore can affect updates, especially when the running average is very small.

QWhich important configuration options does RMSprop expose?

The optimizer accepts parameters or parameter groups and supports options including learning rate, weight decay, momentum, centered, capturable, foreach, maximize, and differentiable. The default learning rate is 1e-2.

Follow-up: When would differentiable be relevant? Use it when the optimizer step itself must participate in autograd, such as differentiating through an optimization procedure.

torch.optim.Optimizer
torch.optim.SGD
torch.optim.Adagrad
torch.optim.Adam
torch.optim.AdamW
torch.optim.lr_scheduler

Source: RMSprop. Checked 2026-10-16.