Keras · Class

layers.Attention

Keras Attention is a Luong-style dot-product attention layer. It converts query-key scores into weights and uses them to combine value vectors.

How it works

Attention compares each query timestep with the key sequence, normalizes those scores into a distribution, and uses it to combine value vectors. With no separate key, the value tensor serves as both key and value; optional masking and score return are supported by the call.

Syntax

keras.layers.Attention(use_scale=False, score_mode="dot", dropout=0.0, seed=None, **kwargs)

Parameters

Name	Type	Description
`use_scale`	`bool`	Whether to multiply attention scores by a trainable scalar.
`score_mode`	`str`	Attention scoring method: `dot` or `concat`.
`dropout`	`float`	Fraction of attention-score units to drop, between 0 and 1.
`seed`	`int or None`	Random seed for dropout.
`**kwargs`	`dict`	Additional base-layer keyword arguments.

Returns

Attention outputs with shape (batch_size, Tq, dim); optionally, attention scores with shape (batch_size, Tq, Tv) when requested by the call.

Key points

This class implements Luong-style dot-product attention.
Pass query and value, and optionally a distinct key, as a list.
Each query timestep receives a score for every key timestep.
Softmax converts the score matrix into normalized attention weights.
The normalized weights mix value vectors to form each output timestep.
The output follows the query sequence length and retains the feature width.

Common mistakes

Omitting the third input makes the value sequence serve as the key sequence automatically.
The dropout setting applies to attention scores, so it regularizes the score-derived distribution rather than the output vectors.
When scores are requested, the second returned tensor contains masked, softmax-normalized attention scores.

Example: Apply dot-product attention

import keras
import numpy as np

query = keras.Input(shape=(3, 4))
value = keras.Input(shape=(5, 4))
output = keras.layers.Attention()([query, value])
model = keras.Model([query, value], output)
result = model([np.random.random((2, 3, 4)), np.random.random((2, 5, 4))])
print(result.shape)

Builds and runs dot-product attention with separate query and value sequences; the value tensor also supplies the keys.

When to use it

Align decoder queries with encoder values

Sequence-to-sequence models

Use one sequence as query, key, and value

Self-attention blocks

Let one sequence select information from another

Cross-modal or cross-sequence fusion

Interview questions

QHow does keras.layers.Attention produce its output?

The layer computes query-key scores with shape (batch_size, Tq, Tv), applies softmax, and uses the resulting distribution to form weighted combinations of value vectors. The output has shape (batch_size, Tq, dim).

Follow-up: What happens when key is omitted? The layer reuses value as the key, producing the common self-attention-style query/value configuration.

QWhat input and output shapes does the layer use?

The layer accepts a list containing query and value, plus an optional key. Query and key have sequence lengths Tq and Tv, while the feature dimension is dim; the output preserves Tq and dim.

Follow-up: What shape do attention scores have? They are (batch_size, Tq, Tv), representing every query-to-key timestep pairing.

QWhich constructor options control scoring and regularization?

use_scale enables a trainable scalar scale for the attention scores, and score_mode selects either dot-product scoring or concatenation-based scoring. The constructor also exposes score dropout and a seed.

Follow-up: Which constructor option controls score regularization? dropout specifies the fraction of attention-score units dropped, while seed controls the random stream for that operation.

keras.layers.AdditiveAttention
keras.layers.MultiHeadAttention
keras.layers.Softmax

Source: Attention layer - Keras. Checked 2026-10-15.