Self-attention
Apply attention within one sequence by passing the same tensor as query, key, and value.
AI & ML · Const
Multi-Head Attention
PyTorch's MultiheadAttention jointly attends to information from multiple representation subspaces. It splits the model dimension across parallel heads, combines their outputs, and supports self-attention or cross-attention.
How it works
MultiheadAttention projects queries, keys, and values into multiple representation subspaces, computes attention independently for each head, concatenates the head outputs, and applies a final projection. With batch_first=True, inputs and outputs use batch, sequence, feature ordering.
Syntax
torch.nn.MultiheadAttention(embed_dim, num_heads, dropout=0.0, bias=True, add_bias_kv=False, add_zero_attn=False, kdim=None, vdim=None, batch_first=False, device=None, dtype=None)Parameters
| Name | Type | Description |
|---|---|---|
embed_dim | int | Total dimension of the model. |
num_heads | int | Number of parallel attention heads. |
dropout | float | Dropout probability on attention weights. |
bias | bool | If True, adds learnable bias to input projections. |
batch_first | bool | If True, input and output tensors use (batch, seq, feature) order. |
query | Tensor | Query embeddings. |
key | Tensor | Key embeddings. |
value | Tensor | Value embeddings. |
attn_mask | Tensor | None | Mask that prevents attention to certain positions. |
Returns
forward returns attn_output and attn_output_weights; weights are averaged across heads by default or returned per head when average_attn_weights=False.
Key points
- Multiple attention heads let the model attend jointly to different representation subspaces.
- The module implements the original Transformer multi-head attention architecture.
- The head outputs are concatenated and passed through an output projection.
- num_heads controls how many attention computations run in parallel.
- The forward call consumes query, key, and value embeddings and returns attention results.
- By default, returned attention weights are averaged across heads; disable averaging to inspect per-head weights.
Common mistakes
- Choose an embed_dim divisible by num_heads so each head receives an integer-sized feature subspace.
- Set batch_first=True when your tensors use batch, sequence, feature order; otherwise the default layout is different.
- A 2D attention mask applies uniformly across batches; use a 3D mask for batch-specific attention restrictions.
Example: Two-head self-attention
import torch
from torch import nn
mha = nn.MultiheadAttention(embed_dim=8, num_heads=2, batch_first=True)
x = torch.randn(1, 4, 8)
y, weights = mha(x, x, x)
print(y.shape, weights.shape)Creates a two-head self-attention layer and applies it to a batch-first sequence.
When to use it
Cross-attention
Use one sequence as query and another as key-value input.
Transformer layers
Build Transformer encoder and decoder components.
Attention visualization
Inspect attention distributions for analysis or debugging.
Related
torch.nn.MultiheadAttentiontorch.nn.functional.multi_head_attention_forwardTransformerEncoderLayerTransformerDecoderLayer