TensorFlow · Class

optimizers.RMSprop

TensorFlow's RMSprop optimizer adapts parameter updates using a discounted moving average of squared gradients. It supports momentum, centered variance normalization, clipping, weight decay, EMA, and gradient accumulation.

How it works

RMSprop tracks a discounted moving average of squared gradients and scales each gradient by the square root of that average. Optional momentum changes the update dynamics, while centered mode also tracks the gradient mean to normalize using an estimated variance.

Syntax

tf.keras.optimizers.RMSprop(learning_rate=0.001, rho=0.9, momentum=0.0, epsilon=1e-07, centered=False, weight_decay=None, clipnorm=None, clipvalue=None, global_clipnorm=None, use_ema=False, ema_momentum=0.99, ema_overwrite_frequency=None, loss_scale_factor=None, gradient_accumulation_steps=None, name='rmsprop', **kwargs)

Parameters

Name	Type	Description
`learning_rate`	`float, LearningRateSchedule, or callable`	Float, learning-rate schedule, or no-argument callable; default 0.001.
`rho`	`float`	Discounting factor for old gradients; default 0.9.
`momentum`	`float`	Momentum coefficient; default 0.0.
`epsilon`	`float`	Small numerical-stability constant; default 1e-7.
`centered`	`bool`	Whether to normalize using estimated variance instead of the uncentered second moment; default False.
`weight_decay`	`float or None`	Optional weight-decay coefficient.
`clipnorm`	`float or None`	Optional per-gradient norm clipping threshold.
`clipvalue`	`float or None`	Optional value clipping threshold.
`global_clipnorm`	`float or None`	Optional global gradient-norm clipping threshold.
`use_ema`	`bool`	Whether to use exponential moving averages of model variables.
`ema_momentum`	`float`	Momentum for the optimizer's exponential moving average; default 0.99.
`ema_overwrite_frequency`	`int or None`	Optional frequency for overwriting model variables with EMA values.

Returns

An RMSprop optimizer instance.

Key points

RMSprop maintains a discounted moving average of squared gradients as its adaptive scaling statistic.
Each gradient is scaled by dividing it by the square root of the tracked average.
Centered mode tracks the gradient mean in addition to squared-gradient information.
learning_rate controls the base step size and defaults to 0.001; it may also be a schedule or callable.
rho controls how strongly older gradient information is discounted and defaults to 0.9.
epsilon prevents numerical problems when the adaptive denominator is very small.

Common mistakes

The momentum option is ordinary momentum; it does not enable Nesterov momentum.
Centered RMSprop can improve training in some cases, but costs additional computation and memory.

Example: Train a model with RMSprop

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(4,)),
    tf.keras.layers.Dense(1)
])
model.compile(
    optimizer=tf.keras.optimizers.RMSprop(learning_rate=0.001),
    loss="mse"
)
model.fit(tf.random.normal((8, 4)), tf.random.normal((8, 1)), epochs=1, verbose=0)

Creates a small model, configures RMSprop as its optimizer, and runs one training epoch.

When to use it

General gradient-based training

Training neural networks with adaptive per-parameter step sizes.

Uneven gradient scales

Training where gradient magnitudes vary substantially across parameters.

Centered normalization

Training experiments that benefit from variance-based normalization.

EMA, clipping, and loss scaling

Large or mixed-precision training setups requiring optimizer-side controls.

Interview questions

QHow does RMSprop adapt updates?

RMSprop maintains a discounted moving average of squared gradients, then divides the current gradient by the square root of that average. This adapts the effective step size per parameter.

Follow-up: What does centered mode add? It also tracks a moving average of gradients and uses it to estimate gradient variance.

QWhat are the important default hyperparameters?

The default learning rate is 0.001, and rho defaults to 0.9. rho controls discounting of older gradient-square information.

Follow-up: How can the learning rate be supplied dynamically? Use a learning-rate schedule or a no-argument callable that returns the value to use.

QHow does gradient accumulation affect updates?

With gradient_accumulation_steps set to an integer, variables are updated only every specified number of steps. The update uses the average of gradients collected since the preceding update.

Follow-up: What happens when accumulation is unset? RMSprop updates normally at each optimizer step.

tf.keras.optimizers.Optimizer
tf.keras.optimizers.schedules.LearningRateSchedule
tf.keras.optimizers.Adam
tf.keras.optimizers.SGD

Source: tf.keras.optimizers.RMSprop | TensorFlow v2.16.1. Checked 2026-10-15.