AI & ML · Const

Self-Attention

Self-attention lets sequence positions integrate information from one another. PyTorch’s TransformerEncoderLayer implements this with a feedforward network and exposes multi-head attention configuration.

How it works

Self-attention lets each position in a sequence combine information from other positions using multi-head attention. In PyTorch, TransformerEncoderLayer packages this self-attention with a feedforward network, normalization, dropout, and residual structure; with batch_first=True, inputs and outputs use batch, sequence, feature order.

Syntax

class torch.nn.modules.transformer.TransformerEncoderLayer(d_model, nhead, dim_feedforward=2048, dropout=0.1, activation=<function relu>, layer_norm_eps=1e-05, batch_first=False, norm_first=False, bias=True, device=None, dtype=None)

Parameters

Name	Type	Description
`d_model`	`int`	Number of expected features in each input element; required.
`nhead`	`int`	Number of heads in the multi-head attention module; required.
`dim_feedforward`	`int`	Feedforward network dimension; default 2048.
`dropout`	`float`	Dropout probability; default 0.1.
`activation`	`string or callable`	Intermediate activation: relu, gelu, or a unary callable; default relu.
`layer_norm_eps`	`float`	Layer-normalization epsilon; default 1e-5.
`batch_first`	`bool`	Whether input and output use batch, sequence, feature order; default false.
`norm_first`	`bool`	Whether normalization occurs before attention and feedforward operations; default false.
`bias`	`bool`	Whether to include bias terms; default true.
`device`	`torch.device or equivalent`	Device for module parameters.
`dtype`	`torch.dtype or equivalent`	Data type for module parameters.

Returns

A tensor with the same sequence-feature shape as the input; with batch_first=True, input and output are (batch, sequence, feature).

Key points

A TransformerEncoderLayer combines self-attention with a feedforward network.
The layer follows the original Transformer encoder architecture.
d_model is the required feature width of each input token.
nhead is the required number of attention heads.
dim_feedforward controls the intermediate feedforward width and defaults to 2048.
dropout sets the dropout probability and defaults to 0.1.

Common mistakes

The default layout is sequence, batch, feature; set batch_first=True when tensors are arranged as batch, sequence, feature.
norm_first changes normalization from after attention and feedforward operations to before them.
The layer supports both regular torch.Tensor inputs and Nested Tensor inputs; derived classes are expected to accept both formats.

Example: Apply a self-attention encoder layer

import torch
from torch import nn

layer = nn.TransformerEncoderLayer(
    d_model=8, nhead=2, batch_first=True
)
x = torch.randn(2, 4, 8)
y = layer(x)
print(y.shape)  # torch.Size([2, 4, 8])

Creates a two-head Transformer encoder layer and applies self-attention plus its feedforward network to a batch-first sequence.

When to use it

Sequence representation learning

Encode token sequences for language or other sequential data.

Transformer encoders

Build an encoder stack from repeated Transformer layers.

Context-aware features

Model relationships between positions in a sequence through multi-head attention.

torch.nn.Transformer
torch.nn.TransformerEncoder
torch.nn.TransformerEncoderLayer
torch.nn.MultiheadAttention

Source: TransformerEncoderLayer — PyTorch 2.14 documentation. Checked 2026-10-15.