TensorFlow · Class

layers.LayerNormalization

tf.keras.layers.LayerNormalization normalizes activations independently for each example across selected axes, typically the feature axis. It can optionally apply trainable scaling and centering after normalization.

How it works

For each example, the layer normalizes activations over the selected axis or axes, making their mean close to zero and standard deviation close to one. Optional trainable gamma and beta variables then scale and shift the normalized result; epsilon stabilizes the computation.

Syntax

tf.keras.layers.LayerNormalization(axis=-1, epsilon=0.001, center=True, scale=True, rms_scaling=False, beta_initializer='zeros', gamma_initializer='ones', beta_regularizer=None, gamma_regularizer=None, beta_constraint=None, gamma_constraint=None, **kwargs)

Parameters

Name	Type	Description
`axis`	`Integer or list/tuple`	Axis or axes over which to normalize; commonly the feature axes. The default is -1.
`epsilon`	`float`	Small value used for numerical stability in the normalization calculation. Default: 0.001.
`center`	`bool`	Whether to add the trainable beta offset. Default: True.
`scale`	`bool`	Whether to multiply by the trainable gamma scale. Default: True.
`rms_scaling`	`bool`	Whether to use RMS scaling. Default: False.
`beta_initializer`	`initializer`	Initializer for beta when center is enabled. Default: 'zeros'.
`gamma_initializer`	`initializer`	Initializer for gamma when scale is enabled. Default: 'ones'.
`beta_regularizer`	`regularizer`	Optional regularizer for beta.
`gamma_regularizer`	`regularizer`	Optional regularizer for gamma.
`beta_constraint`	`constraint`	Optional constraint for beta.
`gamma_constraint`	`constraint`	Optional constraint for gamma.
`**kwargs`	`keyword arguments`	Additional base-layer keyword arguments.

Returns

A layer output with activations normalized over the selected axes, followed by optional trainable scaling and centering.

Key points

Normalization is performed separately for each example rather than using statistics pooled across the batch.
The standard behavior targets a per-example activation mean near zero and standard deviation near one.
axis selects the dimensions whose activations participate in the normalization calculation.
The default axis targets the input's final dimension, which commonly represents features.
The optional affine stage is controlled independently through the scale and center flags.
The normalized value is transformed by multiplying with gamma and adding beta.

Common mistakes

The dimensions covered by axis must be known when the layer is built because they determine the shapes of the trainable parameters.
Setting scale=False can be appropriate when a following linear operation can absorb the scaling.

Example: Normalize features per example

import tensorflow as tf

layer = tf.keras.layers.LayerNormalization(axis=-1)
x = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
y = layer(x)
print(y.numpy())

Creates a layer that normalizes each row independently across its final dimension, then applies it to a small tensor.

When to use it

Normalize feature vectors at each token or timestep without relying on batch statistics.

Transformer and sequence models

Keep normalization behavior independent of the other examples in a batch.

Small or variable batch training

Control activation scale and offset with optional trainable gamma and beta parameters.

Feature stabilization

Interview questions

QHow does LayerNormalization differ from BatchNormalization?

LayerNormalization computes normalization independently for each example, unlike BatchNormalization, which normalizes across a batch. It uses the axes specified by axis, with -1 selecting the final dimension by default.

Follow-up: Why does axis matter for parameter shapes? The trainable gamma and beta span the normalized axes, so those dimensions must be statically defined.

QWhat do center and scale control?

After normalization, the layer can apply a trainable scale gamma and offset beta. With their default initial values—ones for gamma and zeros for beta—the initial affine transformation does not change the normalized values.

Follow-up: When can scale be disabled? If a following linear layer performs the needed scaling, disabling it avoids a redundant trainable scale.

QWhat are the important defaults and configurable axes?

The default axis is -1, epsilon is 0.001, and center and scale are enabled. The layer accepts a single axis or a list or tuple of axes to normalize across.

Follow-up: What does rms_scaling change? It enables RMS-style scaling rather than the standard mean-centering behavior described for ordinary layer normalization.

tf.keras.layers.BatchNormalization
tf.keras.layers.Normalization
tf.keras.layers.UnitNormalization

Source: tf.keras.layers.LayerNormalization | TensorFlow v2.16.1. Checked 2026-10-15.