Keras · Class

optimizers.Lion

Lion is a Keras optimizer that uses a sign-based momentum update instead of Adam-style second-moment adaptation. It can reduce optimizer-state memory while supporting standard options such as weight decay, clipping, EMA, and gradient accumulation.

How it works

Lion updates parameters using the sign of a momentum-based direction rather than Adam-style second-moment adaptation. It maintains momentum information, which gives it a smaller optimizer state than methods that also store second moments. The class also supports common optimizer features such as weight decay, gradient clipping, EMA, loss scaling, and gradient accumulation.

Syntax

keras.optimizers.Lion(learning_rate=0.001, beta_1=0.9, beta_2=0.99, weight_decay=None, clipnorm=None, clipvalue=None, global_clipnorm=None, use_ema=False, ema_momentum=0.99, ema_overwrite_frequency=None, loss_scale_factor=None, gradient_accumulation_steps=None, name=None, **kwargs)

Parameters

Name	Type	Description
`learning_rate`	`float, LearningRateSchedule, or callable`	Learning rate; defaults to 0.001. It may be a float, schedule, or zero-argument callable.
`beta_1`	`float, constant float tensor, or callable`	Rate for combining the current gradient with the first-moment estimate; defaults to 0.9.
`beta_2`	`float, constant float tensor, or callable`	Exponential decay rate for the first-moment estimate; defaults to 0.99.
`weight_decay`	`float or None`	Applies weight decay when set.
`clipvalue`	`float or None`	Clips each gradient by value when set.
`clipnorm`	`float or None`	Clips each gradient by norm when set.
`global_clipnorm`	`float or None`	Clips the global gradient norm when set.
`use_ema`	`bool`	Enables exponential moving average when true.
`ema_momentum`	`float`	EMA momentum; defaults to 0.99.
`ema_overwrite_frequency`	`int or None`	Frequency for overwriting model variables with EMA values.
`loss_scale_factor`	`float or None`	Loss scale factor for mixed-precision training.
`gradient_accumulation_steps`	`int or None`	Number of steps over which to accumulate gradients.

Returns

A Lion optimizer instance.

Key points

Lion is available as a Keras optimizer class implementing the Lion algorithm.
Its optimizer state is comparatively compact because it stores momentum rather than a second moment.
beta_1 controls how the current gradient is combined with the first-moment estimate.
beta_2 controls exponential decay of the first-moment estimate.
Set weight_decay to apply weight decay during optimization.
learning_rate can be a number, a learning-rate schedule, or a zero-argument callable.

Common mistakes

Lion's update uses a sign-based direction, so its behavior differs from adaptive optimizers such as Adam.
When using clipnorm, clipping is applied separately to each weight's gradient.

Example: Compile a model with Lion

import keras

model = keras.Sequential([
    keras.layers.Input(shape=(10,)),
    keras.layers.Dense(1),
])
model.compile(
    optimizer=keras.optimizers.Lion(learning_rate=0.001),
    loss="mse",
)

Creates a Lion optimizer and uses it when compiling a simple Keras model.

When to use it

Memory-conscious optimization

Training a Keras model where optimizer-state memory matters.

Alternative to Adam

Compiling a model with a sign-based momentum optimizer.

Configured optimization

Training workflows requiring built-in decay, clipping, EMA, or accumulation options.

Interview questions

QHow does Lion differ from Adam?

Lion is a stochastic-gradient-descent optimizer that uses the sign operator to control update magnitude. Unlike Adam, it does not rely on second-order moments and therefore only keeps momentum state.

Follow-up: Why can Lion use less optimizer memory than Adam? Adam stores first- and second-moment state, whereas Lion tracks momentum without the second-moment accumulator.

QWhat are Lion's main tunable parameters?

The default learning rate is 0.001, beta_1 defaults to 0.9, and beta_2 defaults to 0.99. These values control the learning rate and the two momentum-related combinations used by the algorithm.

Follow-up: Which parameter should be changed first when updates are unstable? Usually tune learning_rate first, then investigate beta values and clipping if needed.

QHow would you use Lion in a Keras training workflow?

Lion accepts standard Keras optimizer options, including weight decay, gradient clipping, EMA, loss scaling, and gradient accumulation. It can be passed directly to model.compile as the optimizer.

Follow-up: How would you add regularization without changing the loss? Set weight_decay to a numeric value when constructing Lion.

keras.optimizers.Optimizer
keras.optimizers.Adam
keras.optimizers.SGD
keras.optimizers.schedules.LearningRateSchedule

Source: Keras documentation: Lion. Checked 2026-10-15.