Keras · Class

layers.MultiHeadAttention

MultiHeadAttention implements scaled multi-head attention for self-attention or cross-attention. It projects query, key, and value across several heads, combines the attended values, and optionally projects the result to a chosen output shape.

How it works

MultiHeadAttention projects query, key, and value into multiple head representations, compares query with key using scaled dot products, converts those scores into probabilities, and combines the value representations. It concatenates the heads and optionally applies a final projection to produce the requested output width.

Syntax

keras.layers.MultiHeadAttention(num_heads, key_dim, value_dim=None, dropout=0.0, use_bias=True, output_shape=None, attention_axes=None, sliding_window=None, flash_attention=None, kernel_initializer="glorot_uniform", bias_initializer="zeros", kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None, use_gate=False, seed=None, **kwargs)

Parameters

Name	Type	Description
`num_heads`	`int`	Number of attention heads.
`key_dim`	`int`	Size of each query and key attention head.
`value_dim`	`int or None`	Size of each value attention head; defaults to key_dim.
`dropout`	`float`	Dropout probability applied to attention outputs.
`use_bias`	`bool`	Whether the layer's projection layers use bias terms.
`output_shape`	`int, tuple, or None`	Target shape for the final output projection; if omitted, uses the query's final dimension.
`attention_axes`	`int, tuple, or None`	Axes over which attention is applied.
`sliding_window`	`int or None`	Sliding-window size for local attention.
`flash_attention`	`bool or None`	Whether flash attention is enabled, disabled, or selected automatically.
`kernel_initializer`	`initializer`	Initializer for kernel weights.
`bias_initializer`	`initializer`	Initializer for bias weights.
`kernel_regularizer`	`regularizer or None`	Regularizer for kernel weights.

Returns

Returns attention_output with shape (B, T, E), where E is the query input's final dimension when output_shape is None; with return_attention_scores=True, it also returns attention scores.

Key points

The layer implements the multi-headed attention mechanism described by Vaswani and colleagues.
Each input is transformed before attention so every head receives its own query, key, and value representation.
Attention logits come from scaled query-key dot products.
Softmax converts the scaled logits into weights over the source positions.
Each head mixes value vectors using its weights, after which all heads are concatenated.
An optional output_shape applies a final projection to the concatenated multi-head result.

Common mistakes

Passing the same tensor for all three inputs performs self-attention; different key/value tensors implement cross-attention.
With no output_shape, the final dimension follows the query's last dimension; setting output_shape changes the projected output shape.

Example: Cross-attention

import keras
from keras import layers

query = keras.random.normal((2, 5, 16))
context = keras.random.normal((2, 7, 16))
mha = layers.MultiHeadAttention(num_heads=4, key_dim=8)
output = mha(query=query, value=context, key=context)
print(output.shape)  # (2, 5, 16)

The layer lets five query positions attend to seven context positions and returns one output vector per query position.

When to use it

Transformer self-attention

A sequence attends to itself to model relationships between positions.

Encoder-decoder cross-attention

One sequence attends to representations from another sequence.

Spatial or local attention

Attention is restricted to selected dimensions or a local window.

Composable neural-network blocks

The layer is placed in a reusable Keras model or functional graph.

Interview questions

QHow does MultiHeadAttention compute its output?

It splits the projected query, key, and value into num_heads, computes scaled dot products between query and key, normalizes them with softmax, and uses the resulting probabilities to combine value vectors. The heads are concatenated and may receive a final linear projection.

Follow-up: How would you express cross-attention? Pass the decoder states as query and the encoder states as both key and value.

QWhat do num_heads, key_dim, and value_dim control?

num_heads controls how many attention heads are used, while key_dim is the size of each head's query and key representation. value_dim independently controls each head's value width and defaults to key_dim when omitted.

Follow-up: Why might value_dim differ from key_dim? It allows the representation used for value aggregation to have a width distinct from the query-key matching space.

QWhich inputs and controls are available at call time?

The call accepts query, value, and an optional key; if key is omitted, value is used as the key. It can also accept attention_mask and return attention scores when return_attention_scores=True.

Follow-up: What shape should a standard attention mask have? Use a boolean tensor shaped (batch, target length, source length), with true entries marking permitted attention pairs.

keras.layers.Attention
keras.layers.AdditiveAttention
keras.layers.Embedding
keras.layers.LayerNormalization

Source: MultiHeadAttention layer - Keras. Checked 2026-10-15.