Execute a lazy plan
You have composed filters, projections, joins, or aggregations lazily and now need the resulting table.
Polars · Function
collect
Executes a LazyFrame query and materializes its result as a DataFrame. It supports selectable execution engines, query optimization controls, and optional background execution.
How it works
LazyFrame.collect executes the lazy query plan and materializes its output as a DataFrame. It can use automatic, in-memory, streaming, or GPU execution, and background mode returns a handle instead of immediately returning the completed result.
Syntax
LazyFrame.collect(*, engine: EngineType = 'auto', background: bool = False, optimizations: QueryOptFlags = (), **_kwargs: Any) → DataFrame | InProcessQueryParameters
| Name | Type | Description |
|---|---|---|
engine | EngineType = 'auto' | Selects the execution engine; accepts "auto", "in-memory", "streaming", "gpu", or an Engine instance. |
background | bool = False | Runs the query in the background and returns a query handle. |
optimizations | QueryOptFlags = () | Controls which query optimizations are enabled or disabled. |
_kwargs | Any | Additional keyword arguments accepted by the method. |
Returns
Normally returns a DataFrame; with background execution, returns an InProcessQuery handle.
Key points
- Collection turns a lazy query into an eagerly available
DataFrame. - The default behavior enables the available query optimizations.
- The
engineargument chooses how Polars executes the plan. - With no configured affinity, automatic selection defaults to the streaming engine.
- Background collection returns a handle rather than waiting for the query to finish.
- For collecting several lazy frames together, the related
polars.collect_allfunction is provided.
Common mistakes
- Streaming can reduce memory pressure by processing the query in batches, but it is a different execution mode from the in-memory engine.
- With
background=True, the return value is a query handle; use it to retrieve the result or cancel execution. - The GPU engine requires an Nvidia GPU and the
cudf-polarspackage.
Example: Materialize a lazy query
import polars as pl
query = (
pl.LazyFrame({"x": [1, 2, 3]})
.filter(pl.col("x") > 1)
)
result = query.collect()
print(result)Builds a lazy query, then materializes its result as an eager DataFrame.
When to use it
Process in streaming mode
The query is too memory-intensive for ordinary in-memory execution.
Run collection in the background
A long-running query should be started without blocking the caller.
Collect multiple plans
Several lazy plans need coordinated execution.
Interview questions
QWhat does LazyFrame.collect do?
collect materializes a LazyFrame into a DataFrame. Lazy operations build a query plan first, and collection is the step that executes that plan.
Follow-up: How can collection reduce memory pressure? Use engine="streaming", which executes the query in batches.
QWhich execution engines can collect use?
The engine parameter selects query execution: "auto", "in-memory", "streaming", or "gpu". An Engine object can also be supplied for more specific configuration.
Follow-up: When is streaming preferable? Choose it when batch processing can reduce memory pressure or when the workload benefits from streaming execution.
QHow are optimization settings controlled?
By default, query optimizations are enabled. The optimizations parameter can disable individual optimizations through QueryOptFlags.
Follow-up: What does background execution change? With background=True, collection runs in the background and returns a handle that supports fetching the result or cancelling the query.
Related
LazyFrame.explainLazyFrame.profilepolars.collect_allpolars.Config.set_streaming_chunk_size