Maximum-norm regularization
Keep incoming weights from growing beyond a chosen magnitude.
Keras · Module
keras.constraints
Keras constraints are projection callables that enforce rules on layer variables after gradient updates. Built-ins cover norm limits, non-negativity, and unit norms, and layers expose arguments such as kernel_constraint and bias_constraint.
How it works
Keras constraints are callable projection functions applied to model variables after gradient updates. Built-in constraints include maximum norms, bounded norms, non-negative weights, and unit norms; layers expose constraint arguments for supported weights.
Syntax
keras.constraints.MaxNorm(max_value=2, axis=0)
keras.constraints.MinMaxNorm(min_value=0.0, max_value=1.0, rate=1.0, axis=0)
keras.constraints.NonNeg()
keras.constraints.UnitNorm(axis=0)Parameters
| Name | Type | Description |
|---|---|---|
max_value | float | Maximum norm for incoming weights; MaxNorm default is 2. |
axis | int or tuple of ints | Axes over which the constraint computes norms; MaxNorm default is 0. |
min_value, max_value | float | Lower and upper norm bounds for MinMaxNorm; defaults are 0.0 and 1.0. |
rate | float | Interpolation rate used by MinMaxNorm; default is 1.0. |
Returns
A constraint returns a tensor with the same shape and dtype as its input.
Key points
- Constraints project each targeted variable after its gradient update during fit-based training.
- Dense and the listed convolution layers share a consistent constraint interface.
- Use kernel_constraint to constrain a layer's primary weight matrix.
- Use bias_constraint when the bias vector needs its own constraint.
- MaxNorm limits each hidden unit's incoming-weight norm to the configured maximum.
- NonNeg projects weights so their values are non-negative.
Example: Apply a maximum-norm constraint
import keras
from keras import layers
from keras.constraints import max_norm
model = keras.Sequential([
layers.Dense(64, kernel_constraint=max_norm(2.0), input_shape=(32,)),
layers.Dense(10, activation="softmax"),
])
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy")This constrains each hidden unit's incoming kernel weights to a maximum norm of 2 during training.
When to use it
Non-negative weights
Force parameters to remain valid non-negative values.
Bounded norms
Keep per-unit weight norms inside explicit lower and upper limits.
Interview questions
QWhat is a Keras weight constraint and when is it applied?
A constraint is a callable that receives a tensor and returns another tensor with the same shape and dtype. Keras applies it to the target variable after each gradient update when training with fit().
Follow-up: How would you create a custom constraint? Define a subclass of keras.constraints.Constraint and implement its callable behavior to transform the tensor.
QWhat does MaxNorm do, and how is it attached to a layer?
MaxNorm limits the norm of weights connected to each hidden unit to a specified maximum. Its shortcut is keras.constraints.max_norm, and Dense can receive it through kernel_constraint.
Follow-up: How could you constrain biases instead? Pass a suitable constraint through the layer's bias_constraint argument.
QWhich built-in weight constraints does Keras provide?
The documented built-ins include MaxNorm, MinMaxNorm, NonNeg, and UnitNorm. MinMaxNorm keeps norms between lower and upper bounds, NonNeg enforces non-negative weights, and UnitNorm enforces unit norm.
Follow-up: What must a custom constraint preserve? Its returned tensor must have the same shape and dtype as the input tensor.
Related
keras.constraints.Constraintkeras.constraints.MaxNormkeras.constraints.MinMaxNormkeras.constraints.NonNegkeras.constraints.UnitNormkeras.constraints.max_norm