Custom scalar transformation
A transformation depends on Python logic that has no practical native-expression equivalent.
Polars · Module
Applies a custom Python function to each element of a Polars expression. It is flexible but slower than native expressions, so use it only when native operations cannot express the logic.
Expr.map_elements applies a Python callable independently to each value in an expression. In a GroupBy context, the callable receives each pre-aggregated Series instead; declaring return_dtype is recommended, while null handling, function-name passing, and threading are configurable.
Expr.map_elements(function: Callable[[Any], Any], return_dtype: PolarsDataType | DataTypeExpr | None = None, *, skip_nulls: bool = True, pass_name: bool = False, strategy: MapElementsStrategy = 'thread_local') → Expr| Name | Type | Description |
|---|---|---|
function | Callable[[Any], Any] | Python callable applied to values. |
return_dtype | PolarsDataType | DataTypeExpr | None | Output datatype; recommended whenever possible. |
skip_nulls | bool | Whether to avoid invoking the callable for null values; defaults to true. |
pass_name | bool | Whether to pass the Series name to the callable; defaults to false. |
strategy | MapElementsStrategy | Execution mode: 'thread_local' or 'threading'; defaults to 'thread_local'. |
Expr
import polars as pl
df = pl.DataFrame({"value": [1, 2, 3]})
out = df.with_columns(
pl.col("value")
.map_elements(lambda x: x * 2, return_dtype=pl.Int64)
.alias("doubled")
)
print(out)Applies a Python function to each value and declares the resulting integer type.
A transformation depends on Python logic that has no practical native-expression equivalent.
A GroupBy expression needs custom processing of each pre-aggregated group Series.
A callable wraps a library or operation unavailable through Polars expressions.
It applies a custom Python UDF to each element of an expression. In a GroupBy context, each element passed to the function is itself a pre-aggregated Series.
Follow-up: When should map_batches be preferred? Use map_batches when the custom function should receive a whole Series or sequence of Series rather than individual values.
return_dtype specifies the output datatype. The documentation recommends setting it whenever possible; otherwise Polars infers the type from the first non-null value returned.
Follow-up: Why can inference be risky? A first non-null result may not represent later outputs, so an explicit datatype makes the expression's schema predictable.
thread_local runs the Python function on a single thread, while threading runs it on separate threads. Threading is not automatically faster because Python work commonly remains constrained by the GIL.
Follow-up: What condition can make threading worthwhile? The per-element work must be substantial and the function must release the GIL, such as by calling suitable C code.
Expr.map_batchesSeries.map_elementspolars.Exprpl.self_dtype