Normalize channels without batch dependence
Convolutional networks with feature maps
Keras · Class
layers.GroupNormalization
GroupNormalization divides feature channels into groups and normalizes each group using its own mean and variance. Optional beta and gamma parameters then offset and scale the result.
How it works
GroupNormalization partitions the selected feature axis into groups, computes each group's mean and variance, and normalizes values with a small epsilon for numerical stability. Optional trainable beta and gamma parameters then offset and scale the normalized output.
Syntax
keras.layers.GroupNormalization(groups=32, axis=-1, epsilon=0.001, center=True, scale=True, beta_initializer="zeros", gamma_initializer="ones", beta_regularizer=None, gamma_regularizer=None, beta_constraint=None, gamma_constraint=None, **kwargs)Parameters
| Name | Type | Description |
|---|---|---|
groups | int | Number of normalization groups; it must divide the selected input dimension. |
axis | int or list/tuple | Axis or axes over which normalization is performed; defaults to -1. |
epsilon | float | Value added to variance for numerical stability; defaults to 0.001. |
center | bool | Whether to apply the beta offset; defaults to True. |
scale | bool | Whether to apply gamma scaling; defaults to True. |
beta_initializer | initializer | Initializer for beta; defaults to zeros. |
gamma_initializer | initializer | Initializer for gamma; defaults to ones. |
beta_regularizer | regularizer or None | Optional regularizer for beta. |
gamma_regularizer | regularizer or None | Optional regularizer for gamma. |
beta_constraint | constraint or None | Optional constraint for beta. |
gamma_constraint | constraint or None | Optional constraint for gamma. |
**kwargs | keyword arguments | Additional layer keyword arguments. |
Returns
A tensor with the same general shape as the input after group normalization.
Key points
- Normalization statistics are computed separately for each channel group.
- groups accepts values from 1 through the relevant input dimension.
- axis may be an integer or a list or tuple of axes.
- The usual axis selection targets the feature or channel dimension.
- epsilon improves numerical stability during variance-based normalization.
- center enables the beta offset after normalization.
- scale enables multiplicative gamma adjustment.
- The layer accepts inputs with arbitrary rank.
Common mistakes
- Choose a groups value that evenly divides the size of the normalized input axis; otherwise the layer configuration is invalid.
- Set scale=False when a following linear operation already supplies the needed scaling.
Example: Normalize grouped feature channels
import keras
from keras import layers
inputs = keras.Input(shape=(32, 32, 64))
x = layers.GroupNormalization(groups=8)(inputs)
outputs = layers.Conv2D(32, 3, padding="same")(x)
model = keras.Model(inputs, outputs)
model.summary()This model normalizes 64-channel feature maps in eight groups before applying a convolution.
When to use it
Maintain stable per-group statistics
Small-batch training
Place normalization before subsequent nonlinear or convolutional layers
Residual or generative blocks
Interview questions
QHow does GroupNormalization work?
The layer divides channels into groups and calculates mean and variance independently within each group. It then normalizes using those statistics, rather than relying on batch-wide statistics.
Follow-up: How can the grouping be made equivalent to instance-style normalization? Use groups equal to the number of channels, so each channel forms its own group.
QWhat do groups and axis control?
groups controls how many groups are formed, while axis identifies the feature dimension or dimensions to normalize. The normalized input dimension must be divisible by groups.
Follow-up: What configuration normalizes all channels together? Set groups=1, provided the selected axis and input shape match that intended grouping.
QWhat are epsilon, center, and scale used for?
epsilon is added to the variance before division, preventing instability when the variance is zero or very small. center adds beta, and scale multiplies by gamma when enabled.
Follow-up: When might scale be disabled? If the next layer is linear, its own scaling can make gamma unnecessary.
Related
LayerNormalizationBatchNormalizationNormalization