What is the difference among pandas, Polars and NumPy?

NumPy gives you fast, homogeneous n-dimensional arrays and vectorised math. pandas builds labelled DataFrames and Series on top of NumPy-style arrays for tabular analysis. Polars is a separate DataFrame library written in Rust, built on Apache Arrow, with a multi-threaded engine and an optional lazy query optimiser.

Choose NumPy when your data is numeric and uniform, such as matrices, images, signals or model inputs. It is best for linear algebra, broadcasting and element-wise maths. If you need named columns of mixed types, joins or group-by, a DataFrame library fits better.

Pick Polars when data is large, pipelines are slow, or you want to use all CPU cores. Its lazy API can optimise the whole query before running it. pandas is still a fine choice for small to medium data, quick notebook work, or when a library you depend on expects pandas.

Polars is written in Rust, runs most operations across multiple threads and stores data in the columnar Arrow format. Its lazy engine can push down filters, select only needed columns and reorder steps. pandas mostly runs operations one at a time on a single core.

In eager mode each operation runs immediately and returns a DataFrame, like pandas. In lazy mode you build a LazyFrame, for example with pl.scan_parquet, and nothing runs until you call collect(). This lets Polars optimise the full plan, such as skipping unused columns and filtering early.

pandas has an Index that labels rows and drives alignment in joins, arithmetic and selection with loc. Polars has no index at all; rows are identified by position or by column values. This makes Polars behaviour more predictable, but code relying on index alignment must be rewritten.

I would use Polars in lazy mode with pl.scan_csv and run the query with the streaming engine. It reads only needed columns, applies filters early and processes data in batches, so it can handle data larger than memory. pandas would usually need manual chunking with chunksize.

pandas was built on NumPy, and its default numeric columns are stored as NumPy arrays. Many pandas operations call NumPy under the hood, and df.to_numpy() returns an ndarray. Since pandas 2.0, columns can also be backed by PyArrow arrays instead.

Polars uses expressions like pl.col("price") * 2 inside methods such as select, with_columns, filter and group_by().agg. Expressions describe what to compute, not how, so Polars can run them in parallel and optimise them. This replaces much of the pandas style of chained indexing and apply.

A row-wise Python function runs in the interpreter for every row, so you lose vectorised C or Rust speed. In Polars, map_elements also blocks parallelism and optimisation. Prefer built-in vectorised methods or expressions, and use NumPy ufuncs where possible.