Custom model layers
Create a reusable neural-network component whose parameters and child layers are registered by PyTorch.
PyTorch · Function
forward
torch.nn.Module.forward defines the computation performed whenever a module is called. Subclasses override it to transform inputs into outputs, while calling the module instance preserves registered hook behavior.
How it works
Override forward in an nn.Module to describe the computation performed for each call. Invoke the module itself rather than calling forward directly, so PyTorch can run registered hooks around the computation.
Syntax
forward(*input)Parameters
| Name | Type | Description |
|---|---|---|
input | Any | Input values supplied to the module; the concrete types, count, and shapes depend on the subclass implementation. |
Returns
The output produced by the subclass's computation; its type and structure depend on the implementation.
Key points
- Module.forward specifies the computation that runs whenever the module is called.
- A custom module implements its forward method to transform its inputs and produce outputs.
- The example module uses matrix multiplication with a learnable weight and then adds a bias.
- forward is not limited to affine layers; it can express arbitrary tensor computation with multiple inputs or outputs.
- The forward pass applies the module's represented computation to the provided input values.
Common mistakes
- Call the module instance, such as model(input), rather than model.forward(input), because direct forward calls bypass registered hooks.
Example: Implementing a module forward pass
import torch
from torch import nn
class Affine(nn.Module):
def __init__(self, features):
super().__init__()
self.weight = nn.Parameter(torch.randn(features, features))
self.bias = nn.Parameter(torch.zeros(features))
def forward(self, input):
return input @ self.weight + self.bias
model = Affine(3)
output = model(torch.ones(2, 3))
print(output.shape)Defines a module computation in forward and invokes it through the module instance.
When to use it
Forward-pass computation
Express a tensor transformation for a training or inference pipeline.
Model composition
Compose several operations or submodules into one callable model component.
Interview questions
QWhat is the purpose of forward in torch.nn.Module?
Override forward to define the computation performed at every module call. The method can accept the module's inputs and return its output.
Follow-up: Why call the module instead of forward directly? The module call path runs registered hooks, while directly invoking forward silently ignores them.
QWhat can a Module.forward implementation compute?
A module may perform arbitrary computation involving any number of inputs and outputs. Its forward pass applies that computation to the supplied inputs.
Follow-up: How would you make the module participate in training? Use learnable nn.Parameter objects or child modules so their parameters are registered and exposed to the optimizer.
QHow should a custom module be structured and used?
Define forward as an instance method and return the result of the desired tensor operations. Then invoke the module instance, which dispatches to forward and preserves hook behavior.
Follow-up: What should be avoided when hooks matter? Calling module.forward directly, because that route does not execute registered hooks.
Related
torch.nn.Moduletorch.nn.Module.__call__torch.nn.Parametertorch.nn.modules.module.register_module_forward_pre_hook