Sequence-to-sequence models
Use encoder context to construct representations aligned with decoder query positions.
TensorFlow · Class
layers.Attention
A Keras layer implementing Luong-style attention. It scores query-key pairs, normalizes them with softmax, and returns weighted value combinations.
How it works
The layer compares each query vector with every key vector, converts those scores into a softmax distribution across the value sequence, and uses the distribution to form weighted value combinations. If no key is supplied, the value tensor also serves as the key; optional score dropout applies only during training.
Syntax
tf.keras.layers.Attention(use_scale=False, score_mode='dot', dropout=0.0, seed=None, **kwargs)Parameters
| Name | Type | Description |
|---|---|---|
use_scale | bool, default False | Whether to learn a scalar scale applied to attention scores. |
score_mode | string, default "dot" | Score function: "dot" or "concat". |
dropout | float, default 0.0 | Fraction of attention-score units to drop; must be between 0 and 1. |
seed | int or None | Python integer random seed used when dropout is enabled. |
**kwargs | keyword arguments | Additional keyword arguments forwarded to the Keras layer. |
Returns
Attention outputs of shape (batch_size, Tq, dim).
Key points
tf.keras.layers.Attentionimplements single-head Luong-style attention.- Pass
[query, value]or[query, value, key]as the layer input. - Each query position receives a score for every key position.
- Softmax turns the score matrix into normalized attention weights.
- The output is a weighted combination of value vectors rather than a selection of one value.
- The dropout argument specifies the fraction of attention-score units dropped during training.
Common mistakes
- Dropout is controlled by the call-time training flag; inference mode does not add attention-score dropout.
- Only the documented dot-product and concatenation score modes are accepted.
- Omitting the key is valid, but it makes the value sequence determine both matching scores and the weighted output.
Example: Dot-product attention
import tensorflow as tf
query = tf.random.normal([2, 3, 4])
value = tf.random.normal([2, 5, 4])
layer = tf.keras.layers.Attention(use_scale=True)
output = layer([query, value])
print(output.shape) # (2, 3, 4)Creates attention context vectors for each query position from a batch of query and value sequences.
When to use it
Cross-attention
Compute weighted context vectors when query and context sequences have different lengths.
Self-attention
Attend over a sequence using the same tensor as both key and value.
Interview questions
QWhat computation does Attention perform?
It accepts query and value tensors, plus an optional key tensor. Scores have shape (batch_size, Tq, Tv), are normalized with softmax, and weight the values to produce (batch_size, Tq, dim) outputs.
Follow-up: What happens when key is omitted? The value tensor is reused as the key, so the layer performs attention against the value sequence itself.
QHow are attention scores computed?
With score_mode="dot", the layer compares query and key vectors by dot product. With score_mode="concat", it uses the hyperbolic tangent of their concatenation.
Follow-up: How would you choose between the modes? Use dot mode for the standard Luong-style formulation; choose concat when the documented additive-style scoring behavior is desired.
QWhat input and output shapes are required?
The query shape is (batch_size, Tq, dim), while value and optional key shapes are (batch_size, Tv, dim). The resulting output keeps the query length and feature dimension: (batch_size, Tq, dim).
Follow-up: What dimension represents the attended positions? Tv indexes the value/key sequence, and softmax distributes each query position's weights across that dimension.
Related
tf.keras.layers.MultiHeadAttentiontf.keras.layers.Layer