Custom learnable state
Defining trainable model weights or biases directly in a custom Module.
PyTorch · Class
nn.Parameter
torch.nn.Parameter is a Tensor subclass that marks tensor data as a module parameter. When assigned to a Module attribute, it is automatically registered and exposed through the module's parameter iterator.
How it works
nn.Parameter wraps tensor data as a module parameter. When assigned to a Module attribute, PyTorch registers it so module parameter iteration can find it; requires_grad controls whether gradients are tracked for the parameter, independently of the no_grad context's effect on Parameter creation.
Syntax
class torch.nn.parameter.Parameter(data=None, requires_grad=True)Parameters
| Name | Type | Description |
|---|---|---|
data | Tensor | Parameter tensor. |
requires_grad | bool | Whether the parameter requires gradients; optional and defaults to True. |
Returns
A torch.nn.parameter.Parameter, which is a Tensor subclass representing a module parameter.
Key points
- nn.Parameter is a Tensor subclass used to mark tensor data as a module parameter.
- Assigning a Parameter to a Module attribute automatically registers it with that module.
- Registered Parameters are returned during the module's parameters() iteration.
- Assigning an ordinary Tensor attribute does not register it as a module parameter.
- requires_grad specifies whether gradients are required for the Parameter.
Common mistakes
- Creating a Parameter inside torch.no_grad() does not change its default requires_grad behavior.
Example: Register a learnable scalar
import torch
from torch import nn
class ScaledValue(nn.Module):
def __init__(self):
super().__init__()
self.scale = nn.Parameter(torch.ones(()))
model = ScaledValue()
print(list(model.parameters()))Assigning the Parameter as a module attribute makes it appear in the module's parameter iterator.
When to use it
Optimizer parameter discovery
Discovering parameters for optimization through a Module's parameter iterator.
Separating temporary state
Keeping cached or temporary tensor state out of module parameter registration.
Interview questions
QWhat distinguishes nn.Parameter from an ordinary Tensor assigned to a Module?
nn.Parameter is a Tensor subclass intended to represent a module parameter. Assigning it as a Module attribute automatically registers it, unlike assigning an ordinary Tensor.
Follow-up: What happens if temporary state is assigned as a plain Tensor attribute? It is not registered as a module parameter, which avoids treating cached state as learnable model state.
QWhat does requires_grad control for nn.Parameter?
The constructor accepts data and an optional requires_grad flag, whose default is True. The flag determines whether the parameter requires gradients.
Follow-up: How can a non-learnable parameter be created? Pass requires_grad=False, while retaining Parameter registration on the module.
QHow does assigning a Parameter affect module parameter discovery?
A Parameter assigned to a Module attribute is included by the module's parameters() iterator. This lets module-level parameter discovery find it for operations such as optimizer setup.
Follow-up: Why does PyTorch avoid registering every Tensor attribute? Modules may hold temporary state, such as an RNN's last hidden state, that should not become a model parameter.
Related
torch.nn.Moduletorch.nn.Module.parameters()torch.nn.parameter.UninitializedParameter