Avoid completely inactive negative units
A neural network contains negative pre-activation values but should retain gradient flow through them.
Keras · Class
layers.LeakyReLU
LeakyReLU is a Keras activation layer that keeps a small, configurable slope for negative inputs and passes nonnegative inputs through unchanged. It preserves the input shape.
How it works
LeakyReLU transforms negative inputs by multiplying them by a configurable nonnegative slope, while leaving zero and positive inputs unchanged. As a layer, it can be inserted into a model and accepts the usual base-layer keyword arguments.
Syntax
keras.layers.LeakyReLU(negative_slope=0.3, **kwargs)Parameters
| Name | Type | Description |
|---|---|---|
negative_slope | float >= 0.0 | Nonnegative floating-point coefficient used for negative inputs; defaults to 0.3. |
**kwargs | keyword arguments | Additional base-layer keyword arguments, including options such as name and dtype. |
Returns
A tensor with the same shape as the input, after applying the Leaky ReLU transformation.
Key points
- It is a layer form of the Leaky ReLU activation.
- Negative inputs retain a nonzero gradient instead of being fully cut off.
- For negative values, the output is the input multiplied by the configured negative slope.
- Zero and positive values pass through unchanged.
- negative_slope controls the multiplier applied on the negative side.
- If no slope is supplied, the layer uses 0.3.
- The constructor accepts standard layer configuration keywords in addition to negative_slope.
Common mistakes
- negative_slope must be nonnegative; negative values are outside the documented parameter range.
Example: Apply LeakyReLU
import numpy as np
import keras
layer = keras.layers.LeakyReLU(negative_slope=0.5)
x = np.array([-10.0, -5.0, 0.0, 5.0, 10.0])
print(layer(x).numpy())Creates the layer, applies it to negative and nonnegative values, and prints the transformed result.
When to use it
Element-wise model activation
A model needs an activation after a convolution or dense transformation without changing tensor dimensions.
Control negative activation scaling
A model requires a tunable negative-side response rather than a fixed zero output.
Interview questions
QHow does LeakyReLU compute its output?
LeakyReLU is a ReLU variant that retains a small gradient for inactive, negative inputs. For x below zero it returns negative_slope multiplied by x; for x at least zero it returns x.
Follow-up: How does changing negative_slope affect negative activations? It changes their linear scaling and the gradient in that region, while positive outputs remain unchanged.
QWhat is the constructor signature and default slope?
The layer constructor is keras.layers.LeakyReLU(negative_slope=0.3, **kwargs). The slope defaults to 0.3, and kwargs can contain standard layer options such as name and dtype.
Follow-up: How would you give the layer a custom identifier? Pass a name keyword, for example keras.layers.LeakyReLU(name="activation").
QWhat input and output shapes does the layer support?
LeakyReLU preserves the input shape because it applies an element-wise transformation. Its input can have an arbitrary shape, so it can follow dense or convolutional layers.
Follow-up: What shape constraint should be checked when integrating it? The layer itself does not require a particular rank; the surrounding layers determine compatible tensor shapes.
Related
keras.layers.ReLUkeras.layers.PReLUkeras.activations.leaky_relu