Normalize token feature vectors across the embedding dimension.
Transformer embeddings
PyTorch · Class
nn.LayerNorm
torch.nn.LayerNorm normalizes the final dimensions of each input using its mean and variance. It can apply optional learnable per-element scale and bias while preserving the input shape.
How it works
LayerNorm computes the mean and variance over the final dimensions specified by normalized_shape, then standardizes those values and applies optional learnable per-element scale and bias. It uses input statistics in both training and evaluation modes, and the output keeps the input shape.
Syntax
class torch.nn.LayerNorm(normalized_shape, eps=1e-05, elementwise_affine=True, bias=True, device=None, dtype=None)Parameters
| Name | Type | Description |
|---|---|---|
normalized_shape | int or list or torch.Size | Expected trailing input shape; accepts an int, list, or torch.Size. A single int targets the last dimension. |
eps | float | Small value added to the denominator for numerical stability. Default: 1e-5. |
elementwise_affine | bool | Enables learnable per-element weight and bias parameters. Default: True. |
bias | bool | Controls whether an additive bias is learned when elementwise_affine is True. Default: True. |
device | torch.device or None | Optional device for module parameters and buffers. |
dtype | torch.dtype or None | Optional data type for module parameters and buffers. |
Returns
A tensor with the same shape as the input, normalized over the final dimensions specified by normalized_shape.
Key points
- LayerNorm is a module for normalizing activations within each input example.
- A scalar normalized_shape is interpreted as a one-dimensional target shape.
- eps protects the normalization denominator from numerical instability; its default is 1e-5.
- The default affine transform learns one scale and one bias for every normalized element, initialized to identity.
- Unlike layers that track running statistics, LayerNorm derives statistics from the current input regardless of mode.
Common mistakes
- Normalization is applied to the trailing dimensions described by normalized_shape, not automatically across the whole tensor.
- Variance uses the biased estimator, equivalent to torch.var(input, unbiased=False).
- Setting bias=False removes only the additive bias, and matters when elementwise_affine is enabled.
Example: Normalize the last dimension
import torch
from torch import nn
x = torch.randn(20, 5, 10)
layer = nn.LayerNorm(10)
y = layer(x)
assert y.shape == x.shape
print(y.shape)Normalizes each 10-element embedding vector while preserving the input shape.
When to use it
Stabilize intermediate activations without relying on batch-wide statistics.
Deep neural network blocks
Normalize each example independently when batch statistics are undesirable.
Variable batch sizes or sequence lengths
Interview questions
QWhat does normalized_shape control?
normalized_shape identifies the trailing dimensions over which LayerNorm calculates statistics. A single integer is treated as a one-element shape, so LayerNorm(10) normalizes the last dimension when its size is 10.
Follow-up: How does the learned affine transform differ from BatchNorm's usual affine behavior? LayerNorm uses per-element scale and bias, whereas the documentation contrasts BatchNorm's scalar channel or plane parameters.
QHow does LayerNorm compute its output?
The layer computes standardized values using the input mean and variance, adds eps inside the denominator for numerical stability, and then applies optional scale and bias. The variance uses the biased estimator.
Follow-up: What happens in evaluation mode? LayerNorm still computes statistics from the current input because it uses input data in both training and evaluation modes.
QWhat are elementwise_affine and bias used for?
With the default elementwise_affine=True, the module has learnable per-element affine parameters: weights start at ones and biases at zeros. Setting elementwise_affine=False disables those learnable affine parameters.
Follow-up: Can bias be disabled independently? Yes; bias=False suppresses the additive parameter while retaining the scale when elementwise_affine remains enabled.
Related
torch.nn.functional.layer_normtorch.nn.BatchNorm1dtorch.nn.InstanceNorm1dtorch.nn.RMSNorm