Scale feature or embedding vectors to comparable lengths before similarity calculations.
Embedding pipelines
PyTorch · Function
F.normalize
Normalizes vectors in a tensor along a chosen dimension using an Lp norm. It divides by the norm or eps, whichever is larger, with defaults of p=2 and dim=1.
How it works
For every vector taken along dim, F.normalize computes its p-norm and divides the vector by the larger of that norm and eps. By default it uses p=2 over dimension 1, while eps prevents division by zero. The operation normally remains differentiable; supplying out disables differentiation.
Syntax
torch.nn.functional.normalize(input, p=2.0, dim=1, eps=1e-12, out=None)Parameters
| Name | Type | Description |
|---|---|---|
input | Tensor | Input tensor of any shape. |
p | float | Exponent in the norm calculation; default 2. |
dim | int or tuple of ints | Dimension or tuple of dimensions to reduce; default 1. |
eps | float | Small value used to avoid division by zero; default 1e-12. |
out | Tensor, optional | Optional output tensor. Using it makes the operation non-differentiable. |
Returns
Tensor
Key points
- The output divides each vector by the greater of its p-norm and eps.
- Defaults use the Euclidean norm and normalize vectors along dimension 1.
- The input may have any tensor shape; dim determines which vectors are reduced.
- p selects the exponent used to calculate the vector norm.
Common mistakes
- Avoid out when gradients are required, because providing an output tensor disables differentiation.
- eps protects normalization when a vector's norm is zero or extremely small.
Example: Row-wise L2 normalization
import torch
import torch.nn.functional as F
x = torch.tensor([[3.0, 4.0], [0.0, 0.0]])
y = F.normalize(x, p=2.0, dim=1)
print(y)
Normalizes each row with the Euclidean norm; the zero row remains finite because the norm is clamped by eps.
When to use it
Normalize vectors that may contain zeros without division-by-zero failures.
Numerical safety
Use p and dim to normalize with a norm exponent and tensor axis suited to the model.
Custom norms
Interview questions
QHow does F.normalize compute the normalized result?
It normalizes each vector along the selected dimension by dividing by max(norm(v, p), eps). The defaults are p=2, dim=1, and eps=1e-12.
Follow-up: What happens for a zero vector? Its divisor becomes eps, so the output remains a finite zero vector rather than producing division by zero.
QWhat does the dim argument control?
dim identifies the dimension whose vectors are normalized, and it may be an integer or a tuple of integers. The default dimension is 1, so the default behavior is row-like normalization for a typical two-dimensional tensor.
Follow-up: How would you normalize feature vectors stored in the last dimension? Pass dim=-1, assuming that is the feature axis.
QWhat autograd caveat applies to out?
out is an optional destination tensor. When it is supplied, the operation is not differentiable, so omit it in a training path that needs autograd.
Follow-up: What should be changed when gradients are needed? Use the returned tensor instead of an out buffer.
Related
torch.linalg.vector_normtorch.nn.functional.normalizetorch.nn.functional.layer_normtorch.nn.functional.rms_norm