Create previous-value features
Feature engineering
Polars · Module
.shift
Shifts expression values by a chosen number of row positions. Positive offsets behave like SQL LAG, negative offsets like LEAD, and fill_value can replace boundary nulls.
How it works
Expr.shift returns an expression whose values are moved by a specified number of row positions. Positive offsets move values forward and create leading nulls, while negative offsets move values in the opposite direction; fill_value can replace the nulls introduced by the shift.
Syntax
Expr.shift(*n: int | IntoExprColumn = 1*, ***, *fill_value: IntoExpr | None = None*) → ExprParameters
| Name | Type | Description |
|---|---|---|
n | int | IntoExprColumn | Number of indices to shift forward; negative values shift in the opposite direction. |
fill_value | IntoExpr | None | Scalar value used to fill nulls produced by the shift. |
Returns
An Expr containing the shifted values.
Key points
- The method shifts values by row index positions rather than changing the underlying row order.
- The expression accepts an integer or column expression offset and optionally a fill value.
- n specifies how many positions the values move in the forward direction.
- Negative n reverses the direction of the shift, enabling look-ahead calculations.
- A positive shift corresponds conceptually to SQL LAG.
- fill_null is the related operation to consider when null handling should be performed separately.
Common mistakes
- A negative offset looks ahead rather than behind, so use it for lead-style calculations.
- The optional fill_value replaces nulls created at the shifted boundary; without it, those positions remain null.
Example: Lag and lead values
import polars as pl
df = pl.DataFrame({"a": [10, 20, 30, 40]})
result = df.with_columns(
previous=pl.col("a").shift(),
next=pl.col("a").shift(-1),
)
print(result)Creates lagged and lead values from column a; boundary positions remain null.
When to use it
Compare current rows with future or past rows
Time-series analysis
Replace shift-generated boundary nulls
Data cleaning
Build lag and lead expressions inside with_columns
Window-style calculations
Interview questions
QWhat does pl.col("x").shift() do?
Expr.shift moves expression values by row position. The default offset is 1, which produces a lag-like result and nulls at the beginning; a negative offset produces a lead-like result.
Follow-up: How would you avoid null boundary values? Pass fill_value to Expr.shift, or apply fill_null when a separate replacement step is preferable.
QHow does the sign of n affect the result?
The n argument controls the displacement and defaults to 1. Positive n shifts forward, while negative n shifts in the opposite direction.
Follow-up: What happens when n is -2? Values are taken from two positions ahead, and the final two positions become null unless filled.
QHow can boundary nulls be replaced?
fill_value supplies the scalar used for null values produced by shifting. It is optional and is applied to the resulting null positions.
Follow-up: When might fill_null be useful instead? Use it when null replacement should be expressed as a separate operation or use a more general expression.
Related
polars.Expr.fill_nullpolars.Series.shiftpolars.DataFrame.shiftpolars.Expr.list.shift