PyTorch · Class

nn.Transformer

nn.Transformer is a reference encoder-decoder implementation of the original Transformer architecture. It processes source and target feature sequences and returns decoder outputs.

How it works

nn.Transformer combines encoder and decoder stacks based on the original Transformer architecture. It accepts source and target feature sequences, applies attention and feed-forward processing, and returns decoder output; batch_first changes sequence layout without changing the feature dimension.

Syntax

torch.nn.modules.transformer.Transformer(d_model=512, nhead=8, num_encoder_layers=6, num_decoder_layers=6, dim_feedforward=2048, dropout=0.1, activation=<function relu>, custom_encoder=None, custom_decoder=None, layer_norm_eps=1e-05, batch_first=False, norm_first=False, bias=True, device=None, dtype=None)

Parameters

Name	Type	Description
`d_model`	`int`	Number of expected features in encoder and decoder inputs; default 512.
`nhead`	`int`	Number of attention heads; default 8.
`num_encoder_layers`	`int`	Number of encoder sub-layers; default 6.
`num_decoder_layers`	`int`	Number of decoder sub-layers; default 6.
`dim_feedforward`	`int`	Feed-forward network width; default 2048.
`dropout`	`float`	Dropout probability; default 0.1.
`activation`	`str \| Callable[[Tensor], Tensor]`	String relu or gelu, or a unary callable.
`custom_encoder`	`Optional[Any]`	Optional custom encoder module.
`custom_decoder`	`Optional[Any]`	Optional custom decoder module.
`layer_norm_eps`	`float`	LayerNorm epsilon; default 1e-5.
`batch_first`	`bool`	If true, use batch, sequence, feature layout; default false.
`norm_first`	`bool`	If true, normalize before attention and feed-forward operations; default false.

Returns

A tensor containing the decoder output, with the target sequence length and the model feature dimension.

Key points

The class provides a standard encoder-decoder Transformer module.
d_model specifies the feature width expected by both encoder and decoder inputs.
nhead controls how many attention heads each multi-head attention module uses.
dim_feedforward sets the hidden width of the intermediate feed-forward networks.
norm_first selects pre-normalization when enabled; otherwise normalization occurs after those operations.

Example: Run a batch-first Transformer

import torch
from torch import nn

model = nn.Transformer(d_model=32, nhead=4, batch_first=True)
src = torch.randn(2, 5, 32)
tgt = torch.randn(2, 3, 32)
out = model(src, tgt)
print(out.shape)  # torch.Size([2, 3, 32])

Creates a batch-first encoder-decoder Transformer and runs source and target sequences with matching embedding dimensions.

When to use it

Foundational architecture experiments

Teaching or learning the original Transformer design.

Encoder-decoder modeling

Sequence-to-sequence models with separate source and target representations.

Prototype sequence models

Small controlled experiments where a reference implementation is sufficient.

Interview questions

QWhat is nn.Transformer intended for?

It is a reference implementation of the original Transformer architecture with encoder and decoder stacks. Its documentation notes that it has limited features compared with newer Transformer architectures.

Follow-up: Why might a newer architecture be preferable? The documented intent is foundational understanding, so newer Transformer variants may provide features absent here.

QHow does batch_first affect tensor shapes?

The default layout is sequence-first: inputs and outputs use (sequence, batch, feature). With batch_first=True, they use (batch, sequence, feature).

Follow-up: What must remain consistent? Source, target, and mask shapes must follow the selected layout and the documented shape conventions.

QWhat are the main architectural constructor parameters?

The constructor takes d_model, nhead, encoder and decoder layer counts, feed-forward width, dropout, activation, optional custom stacks, normalization settings, and device or dtype options. The forward call returns the decoder output tensor.

Follow-up: What constraint applies to d_model and nhead? d_model must be divisible by nhead so features can be partitioned among attention heads.

torch.nn.TransformerEncoder
torch.nn.TransformerDecoder
torch.nn.TransformerEncoderLayer
torch.nn.TransformerDecoderLayer
torch.nn.MultiheadAttention

Source: Transformer — PyTorch documentation. Checked 2026-10-16.