Step decay
Reduce the optimizer's rate after a chosen number of epochs.
TensorFlow · Class
callbacks.LearningRateScheduler
A Keras callback that changes the optimizer's learning rate from a user-supplied function. The function is evaluated at the start of each epoch using the epoch index and current rate.
How it works
At the beginning of each epoch, TensorFlow calls the supplied schedule with the zero-based epoch number and optimizer's current learning rate. The returned floating-point value becomes the optimizer's learning rate for that epoch; verbose can optionally print update messages.
Syntax
tf.keras.callbacks.LearningRateScheduler(schedule, verbose=0)Parameters
| Name | Type | Description |
|---|---|---|
schedule | function | A function receiving the zero-based epoch index and current learning rate, returning the new learning rate as a float. |
verbose | int | Controls logging: 0 is quiet and 1 logs update messages. |
Returns
A LearningRateScheduler callback object.
Key points
- The callback evaluates the schedule at the beginning of every epoch.
- The schedule function is supplied when constructing the callback.
- Each schedule call receives both the current epoch and current learning rate.
- The returned value is applied to the model optimizer.
Common mistakes
- The schedule receives zero-based epoch indices, so the first epoch is epoch 0.
- Set verbose to 1 only when per-epoch learning-rate update messages are wanted.
Example: Epoch-based learning-rate schedule
import numpy as np
import tensorflow as tf
from tensorflow import keras
def scheduler(epoch, lr):
return lr if epoch < 2 else lr * 0.5
model = keras.Sequential([keras.layers.Dense(1, input_shape=(2,))])
model.compile(optimizer=keras.optimizers.SGD(learning_rate=0.1), loss="mse")
callback = keras.callbacks.LearningRateScheduler(scheduler, verbose=1)
model.fit(np.zeros((4, 2)), np.zeros((4, 1)), epochs=3, callbacks=[callback], verbose=0)This keeps the current rate for two epochs, then halves it at the start of each later epoch.
When to use it
Exponential decay
Multiply the current rate by a factor as training progresses.
Warm-up and staged policies
Keep or alter the rate conditionally based on the epoch number.
Interview questions
QWhat does LearningRateScheduler do?
It is a Callback that changes the optimizer's learning rate at epoch boundaries. TensorFlow calls the schedule with the current zero-based epoch and learning rate, then applies the returned value.
Follow-up: How would you preserve the rate for an initial warm-up period? Return the supplied lr while epoch is below the warm-up threshold, then compute and return the new rate.
QWhat function signature does schedule require?
The schedule must accept two arguments: an integer epoch index and the current learning rate. It must return the replacement learning rate as a float.
Follow-up: What happens if the schedule ignores lr? It can still return an independently calculated value, but it then cannot build a schedule relative to the current rate.
QWhen is the learning rate updated?
The callback runs at the start of every epoch, before that epoch's training begins. The resulting value is assigned to the optimizer's learning rate.
Follow-up: Why might a batch-level policy need another design? This callback is epoch-based, so a batch-level policy should use a callback hook or schedule mechanism that updates at batch boundaries.
Related
tf.keras.callbacks.Callbacktf.keras.optimizers.schedules.LearningRateScheduletf.keras.callbacks.ReduceLROnPlateau