Nonlinear intermediate activation
Transformer and feed-forward networks
PyTorch · Class
nn.GELU
nn.GELU is a Gaussian Error Linear Units activation module. It supports the default formulation or a tanh-based approximation and preserves the input shape.
How it works
GELU transforms each input element using the Gaussian cumulative distribution function. Setting approximate="tanh" uses the documented tanh-based estimate instead of the default exact formulation; the module preserves the input shape.
Syntax
class torch.nn.GELU(approximate='none')Parameters
| Name | Type | Description |
|---|---|---|
approximate | str, optional | Selects the GELU approximation algorithm; accepted values are "none" and "tanh", with "none" as the default. |
Returns
A Tensor with the same shape as the input.
Key points
- nn.GELU is a neural-network activation module implementing the Gaussian Error Linear Units function.
- The default formulation multiplies each value by the Gaussian cumulative distribution function evaluated at that value.
- The tanh option selects an approximation rather than the default GELU computation.
- Inputs can be scalars, vectors, batches, or higher-dimensional tensors.
- The module's forward operation returns a Tensor.
Common mistakes
- GELU does not alter tensor dimensions: its output has the same shape as its input.
Example: Apply GELU to a tensor
import torch
from torch import nn
m = nn.GELU()
x = torch.randn(2)
y = m(x)
print(y.shape)Creates the default GELU module, applies it element-wise to a two-element tensor, and prints the unchanged output shape.
When to use it
Element-wise activation across arbitrary dimensions
Batched or sequence tensor processing
Optional tanh approximation
Performance-sensitive model variants
Interview questions
QWhat does nn.GELU compute?
nn.GELU is an activation module that applies GELU element-wise. With the default setting, it computes x times the Gaussian cumulative distribution function at x.
Follow-up: What does approximate="tanh" change? It replaces the default formulation with the documented tanh-based estimate involving x and x cubed.
QWhat values can approximate take?
The approximate argument selects the algorithm and accepts either "none" or "tanh". It is optional and defaults to "none".
Follow-up: How would you select the approximation? Construct the module as nn.GELU(approximate="tanh").
QWhat are GELU's input and output shapes?
The input may have any number of dimensions, represented by (*). The output retains exactly the same shape, while forward returns a Tensor.
Follow-up: Can it process a batch or sequence tensor? Yes; those dimensions are permitted because the documented input shape allows any number of dimensions.
Related
torch.nn.functional.gelutorch.nn.modules.activation.GELUtorch.nn.TransformerEncoderLayer