PyTorch · Class

lr_scheduler.CosineAnnealingLR

CosineAnnealingLR applies cosine annealing to every optimizer parameter group. It gradually moves learning rates toward eta_min over T_max scheduler iterations.

How it works

CosineAnnealingLR updates every optimizer parameter group according to a cosine schedule. Starting from each group’s initial learning rate, it recursively moves toward eta_min across T_max iterations; unlike the warm-restart variant, this scheduler does not restart the cycle.

Syntax

class torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max, eta_min=0.0, last_epoch=-1)

Parameters

Name	Type	Description
`optimizer`	`torch.optim.Optimizer`	Optimizer whose parameter-group learning rates are scheduled.
`T_max`	`int`	Maximum number of iterations.
`eta_min`	`float`	Minimum learning rate; default: 0.
`last_epoch`	`int`	Index of the last epoch; default: -1.

Returns

A torch.optim.lr_scheduler.CosineAnnealingLR scheduler object.

Key points

The class adjusts the learning rate independently for every optimizer parameter group.
T_max defines how many scheduler iterations the annealing interval spans.
eta_min is the lower learning-rate boundary and defaults to 0.0.
last_epoch identifies the most recently completed epoch when resuming or initializing scheduler state.
Its recursive update approximates the cosine schedule associated with SGDR.

Common mistakes

The schedule is recursive, so each update depends on the current learning rate rather than only directly evaluating a closed-form expression.

Example: Cosine-annealed optimizer learning rate

import torch

model = torch.nn.Linear(4, 2)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
    optimizer, T_max=10, eta_min=0.001
)

for epoch in range(10):
    optimizer.zero_grad()
    loss = model(torch.randn(8, 4)).square().mean()
    loss.backward()
    optimizer.step()
    scheduler.step()
    print(epoch + 1, scheduler.get_last_lr())

Creates an SGD optimizer and lowers its learning rate with cosine annealing over 10 scheduler steps.

When to use it

Gradually reduce the learning rate

Training a model with a fixed optimization horizon

Anneal to a nonzero floor

Maintaining a small update scale late in training

Coordinate group learning rates

Scheduling several parameter groups together

Interview questions

QWhat does CosineAnnealingLR control?

CosineAnnealingLR sets each optimizer parameter group’s learning rate with cosine annealing. T_max specifies the maximum number of iterations, and eta_min is the lower bound, defaulting to 0.

Follow-up: How would you obtain the current values? Use the scheduler’s inherited get_last_lr() method after stepping the scheduler.

QWhat does T_max mean?

T_max is the maximum number of scheduler iterations over which the cosine schedule is applied. It is an integer and should match the intended unit of scheduler.step(), such as epochs or batches.

Follow-up: What happens if scheduler.step() is called per batch while T_max was chosen in epochs? The schedule completes much sooner because each call advances the iteration count.

QHow does eta_min affect the schedule?

eta_min is the minimum learning rate for each parameter group, and its default is 0.0. The schedule starts from the optimizer groups’ initial learning rates and anneals toward that floor.

Follow-up: How can a nonzero final learning rate be retained? Pass the desired floor through eta_min when constructing the scheduler.

torch.optim.lr_scheduler.CosineAnnealingWarmRestarts
torch.optim.lr_scheduler.LRScheduler
torch.optim.Optimizer

Source: CosineAnnealingLR — PyTorch 2.13 documentation. Checked 2026-10-16.