TensorFlow · Class

optimizers.SGD

tf.keras.optimizers.SGD performs gradient descent with an optional momentum accumulator. It supports configurable learning rates, Nesterov momentum, gradient clipping, weight decay, EMA, and gradient accumulation.

How it works

SGD updates trainable variables using their gradients, subtracting a learning-rate-scaled gradient from each variable. A nonzero momentum value maintains a velocity-like accumulator to accelerate movement in consistent directions and reduce oscillation; optional Nesterov momentum, clipping, decay, EMA, and gradient accumulation modify the optimization process.

Syntax

tf.keras.optimizers.SGD(learning_rate=0.01, momentum=0.0, nesterov=False, weight_decay=None, clipnorm=None, clipvalue=None, global_clipnorm=None, use_ema=False, ema_momentum=0.99, ema_overwrite_frequency=None, loss_scale_factor=None, gradient_accumulation_steps=None, name='SGD', **kwargs)

Parameters

Name	Type	Description
`learning_rate`	`float, keras.optimizers.schedules.LearningRateSchedule, or callable`	Learning-rate value, schedule, or no-argument callable; defaults to 0.01.
`momentum`	`float`	Nonnegative momentum coefficient; defaults to 0.0.
`nesterov`	`bool`	Whether to use Nesterov momentum; defaults to False.
`weight_decay`	`float or None`	Optional weight-decay factor.
`clipnorm`	`float or None`	Per-variable gradient norm clipping threshold.
`clipvalue`	`float or None`	Gradient value clipping threshold.
`global_clipnorm`	`float or None`	Global gradient norm clipping threshold.
`use_ema`	`bool`	Whether to apply exponential moving averages; defaults to False.
`ema_momentum`	`float`	EMA momentum coefficient; defaults to 0.99.
`ema_overwrite_frequency`	`int or None`	How often EMA values overwrite model variables.
`loss_scale_factor`	`float or None`	Optional loss scaling factor.
`gradient_accumulation_steps`	`int or None`	Number of steps over which gradients are accumulated.

Returns

An SGD optimizer instance.

Key points

SGD is Keras's optimizer for ordinary gradient descent, optionally augmented with momentum.
With momentum, the optimizer stores a velocity that combines the prior velocity with the current gradient step.
The momentum update applies the resulting velocity directly to the variable.
The learning rate can be fixed, scheduled, or supplied dynamically by a callable.
Setting use_ema enables exponential moving averages for optimizer-managed variable updates.

Common mistakes

Nesterov behavior is disabled by default and must be explicitly enabled with nesterov=True.
clipnorm clips each weight's gradient independently, so it is different from global_clipnorm.

Example: Train with SGD

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Dense(1, input_shape=(1,))
])
optimizer = tf.keras.optimizers.SGD(learning_rate=0.1, momentum=0.9)
model.compile(optimizer=optimizer, loss="mse")
model.fit([[0.0], [1.0]], [[0.0], [1.0]], epochs=2, verbose=0)

Creates an SGD optimizer with a fixed learning rate and momentum, then uses it to train a small Keras model.

When to use it

Baseline neural-network training

Train a Keras model with standard first-order updates.

Momentum-based optimization

Reduce oscillation or accelerate progress along consistent gradient directions.

Gradient clipping

Control unstable or excessively large gradients.

Scheduled learning rates and EMA

Use a changing step size or maintain smoothed model weights.

Interview questions

QWhat update does SGD perform?

With momentum set to zero, SGD applies the update w = w - learning_rate * g. With momentum above zero, it maintains a velocity and updates that velocity using the previous velocity and the current gradient before changing w.

Follow-up: How would you provide a changing learning rate? Pass a LearningRateSchedule instance or a no-argument callable as learning_rate.

QWhat forms can learning_rate take?

The learning_rate controls the size of each gradient step and defaults to 0.01. It may be a floating-point value, a LearningRateSchedule, or a no-argument callable returning the current rate.

Follow-up: What should a schedule depend on? A schedule is evaluated using the optimizer's current step through its schedule interface.

QHow do momentum and Nesterov momentum differ in this API?

Nesterov momentum is an optional variant of momentum controlled by nesterov. It defaults to False, so ordinary momentum is used unless explicitly enabled.

Follow-up: What other stabilization control is available? Gradient clipping can be configured with clipnorm, clipvalue, or global_clipnorm.

tf.keras.optimizers.Optimizer
tf.keras.optimizers.schedules.LearningRateSchedule
tf.keras.optimizers.schedules.ExponentialDecay
tf.keras.optimizers.schedules.PiecewiseConstantDecay

Source: tf.keras.optimizers.SGD | TensorFlow v2.16.1. Checked 2026-10-15.