Process each token using context from the same sequence.
Transformer self-attention
PyTorch · Class
nn.MultiheadAttention
torch.nn.MultiheadAttention implements Transformer multi-head attention, combining attention from multiple representation subspaces. It supports configurable head count, dropout, projection biases, key/value dimensions, layouts, devices, and dtypes.
How it works
The module projects query, key, and value representations into multiple heads, computes attention independently in each representation subspace, concatenates the head outputs, and applies an output projection. It can return both the transformed sequence and attention weights, while selecting optimized scaled-dot-product implementations when supported.
Syntax
class torch.nn.modules.activation.MultiheadAttention(embed_dim, num_heads, dropout=0.0, bias=True, add_bias_kv=False, add_zero_attn=False, kdim=None, vdim=None, batch_first=False, device=None, dtype=None)Parameters
| Name | Type | Description |
|---|---|---|
embed_dim | int | Total dimension of the model. |
num_heads | int | Number of parallel attention heads. |
dropout | float | Dropout probability on attention output weights; default 0.0. |
bias | bool | Whether to add bias to input and output projection layers; default True. |
add_bias_kv | bool | Whether to add bias to the key and value sequences; default False. |
add_zero_attn | bool | Whether to add a zero-attention token; default False. |
kdim | int | None | Total key feature dimension; default None, which uses embed_dim. |
vdim | int | None | Total value feature dimension; default None, which uses embed_dim. |
batch_first | bool | Whether inputs and outputs use (batch, sequence, feature); default False. |
device | torch.device | None | Device for module parameters. |
dtype | torch.dtype | None | Data type for module parameters. |
Returns
The forward call returns the attention output and attention weights; tensor shapes depend on batch_first, query/key/value lengths, and whether weights are averaged across heads.
Key points
- MultiheadAttention combines several attention heads so the model can focus on distinct representation subspaces at the same time.
- The class implements the original Transformer multi-head attention architecture.
- num_heads controls how many independent attention heads operate in parallel.
- dropout specifies the probability applied to the attention output weights and defaults to no dropout.
- bias determines whether the input and output projection layers include bias parameters.
- kdim sets the key feature width, independently of the model embedding width when provided.
Common mistakes
- Choose an embedding dimension divisible by the number of heads so each head receives an integral feature width.
- Set batch_first=True when tensors use batch, sequence, feature ordering; otherwise the documented ordering is sequence, batch, feature.
- The specialized inference path has strict conditions, so do not assume Nested Tensor acceleration applies to every evaluation call.
Example: Batched self-attention
import torch
from torch import nn
mha = nn.MultiheadAttention(embed_dim=16, num_heads=4, batch_first=True)
x = torch.randn(2, 5, 16)
out, weights = mha(x, x, x)
print(out.shape, weights.shape)Creates a four-head self-attention layer and applies it to a batch of two sequences, each with five 16-dimensional tokens.
When to use it
Use one sequence as queries and another as keys and values.
Encoder-decoder attention
Combine several learned attention subspaces into a single projected representation.
Sequence representation learning
Interview questions
QWhat does MultiheadAttention compute?
It divides the model embedding dimension among parallel heads, performs attention in each learned query/key/value subspace, concatenates the results, and applies an output projection. This lets the layer jointly attend to different representation subspaces.
Follow-up: When can the layer use an optimized attention implementation? It uses optimized scaled-dot-product attention when possible, and inference may use a stricter Nested Tensor fast path.
QWhat constraints relate embed_dim and num_heads?
The required constructor arguments are embed_dim, the total model feature dimension, and num_heads, the number of parallel heads. Each head has width embed_dim // num_heads, so the dimensions must divide cleanly.
Follow-up: How can key and value widths differ from the query/model width? Supply kdim and vdim; when omitted, each defaults to embed_dim.
QHow does batch_first affect tensor layout?
With batch_first=True, inputs and outputs use (batch, sequence, feature) layout. The module returns an attention output and, when requested by the forward call, attention weights.
Follow-up: What shape can a mask have? A two-dimensional mask is broadcast across batches, while a three-dimensional mask can specify different entries per batch.
Related
torch.nn.functional.multi_head_attention_forwardtorch.nn.functional.scaled_dot_product_attentiontorch.nn.Transformertorch.nn.TransformerEncoderLayer