Weight regularization
Reduce excessive parameter magnitudes during model training.
Keras · Module
keras.regularizers
Keras regularizers add scalar penalties for layer weights or activity to the loss used during optimization. The module provides built-in L1, L2, combined L1/L2, and orthogonality-based regularizers, while also accepting custom callable functions.
How it works
A regularizer is a callable object or function that maps a weight or activation tensor to a scalar penalty. Keras adds penalties produced by layer regularizers to the model loss during optimization, and a layer's collected penalties can be inspected through layer.losses after the layer has processed inputs.
Syntax
keras.regularizers.Regularizer()
keras.regularizers.L1(l1=0.01)
keras.regularizers.L2(l2=0.01)
keras.regularizers.L1L2(l1=0.0, l2=0.0)
keras.regularizers.OrthogonalRegularizer(factor=0.01, mode="rows")Parameters
| Name | Type | Description |
|---|---|---|
l1 | float | Coefficient controlling the L1 penalty. |
l2 | float | Coefficient controlling the L2 penalty. |
factor | float | Coefficient for OrthogonalRegularizer. |
mode | string | Whether OrthogonalRegularizer operates on rows or columns; supported values are "rows" and "columns". |
Returns
A scalar tensor representing the regularization penalty for the supplied tensor.
Key points
- Regularization can target trainable parameters or a layer's output activity.
- The penalty becomes part of the loss optimized during training.
- Penalties are attached and accumulated separately for each layer.
- Many common layers expose kernel_regularizer, bias_regularizer, and activity_regularizer.
- L2 penalizes the squared magnitude of the selected tensor.
- L1L2 combines absolute-value and squared-value penalties.
Common mistakes
- Inspect layer.losses only after the layer has been called on inputs; that is when its regularization penalties are available.
- Custom regularizers must accept the relevant tensor and return one scalar tensor.
Example: Apply an L2 kernel regularizer
from keras import layers, regularizers
layer = layers.Dense(
4,
kernel_regularizer=regularizers.L2(l2=0.01),
)
_ = layer([[1.0, 2.0, 3.0]])
print(layer.losses)A Dense layer receives an L2 penalty on its kernel; after the layer is called, the resulting penalty is available through layer.losses.
When to use it
Activity regularization
Penalize the output of a layer rather than its kernel or bias.
Orthogonality constraints
Encourage feature vectors or matrix dimensions to become mutually orthogonal.
Custom regularization
Define a domain-specific penalty not supplied by the built-in classes.
Interview questions
QWhat does a Keras regularizer do?
Regularizers compute penalties on layer parameters or activity, and Keras sums those penalties into the optimized loss. Built-in choices include L1, L2, L1L2, and OrthogonalRegularizer.
Follow-up: How would you verify that a layer contributed a penalty? Call the layer on inputs, then inspect layer.losses.
QHow do L1 and L2 differ?
L1 uses the sum of absolute values multiplied by l1, while L2 uses the sum of squared values multiplied by l2. L1L2 combines both penalties.
Follow-up: Which built-in regularizer encourages vectors to be orthogonal? OrthogonalRegularizer, whose mode selects rows or columns.
QWhere can regularizers be applied?
Layer APIs commonly expose kernel_regularizer, bias_regularizer, and activity_regularizer. These attach penalties respectively to the kernel, bias, or layer output.
Follow-up: Can a regularizer be custom? Yes; any function accepting a weight matrix and returning a scalar tensor can serve as one.
Related
RegularizerL1L2L1L2OrthogonalRegularizerkeras.layers.Layer.losses