Skip to content
masterclassBuddyTech Buzz5 fresh blogs · 5:00 AM IST
Start learning

Polars · Module

df.unique

Removes duplicate rows from a DataFrame. You can define duplicates with selected columns or expressions, choose which duplicate to retain, and optionally preserve input order.

How it works

Call DataFrame.unique to remove duplicate rows. By default, every column defines duplication; use subset to define the comparison columns, keep to choose the retained row policy, and maintain_order when output order must match the input.

Syntax

DataFrame.unique(subset: IntoExpr | Collection[IntoExpr] | None = None, *, keep: UniqueKeepStrategy = 'any', maintain_order: bool = False) → DataFrame

Parameters

NameTypeDescription
subsetIntoExpr | Collection[IntoExpr] | NoneColumn name(s), selector(s), or expressions used to identify duplicate rows; None considers all columns.
keepUniqueKeepStrategyDuplicate-row retention policy: first, last, any, or none.
maintain_orderboolWhether to preserve the original DataFrame order; this is more expensive and disables streaming.

Returns

DataFrame with unique rows.

Key points

  • The method produces a new DataFrame containing one or zero representatives according to the duplicate policy.
  • Leaving subset unset makes the complete row determine whether two rows are duplicates.
  • subset can be a single column, multiple selectors, or expressions for custom duplicate keys.
  • Choose keep="none" to discard every row belonging to a duplicated group.
  • keep="first" retains the first representative encountered for each duplicate group.
  • keep="last" retains the last representative encountered for each duplicate group.

Common mistakes

  • With keep="any", the retained duplicate row is deliberately unspecified, so use first or last when selection matters.
  • maintain_order=True prevents execution with the streaming engine and costs more to compute.

Example: Deduplicate by selected columns

import polars as pl

df = pl.DataFrame({
    "user": ["a", "a", "b"],
    "score": [10, 12, 8],
})

print(df.unique(subset="user", keep="first", maintain_order=True))

Keeps the first row for each distinct user while preserving the input order.

When to use it

Remove duplicate rows

A dataset contains repeated complete records and only one representative is needed.

Deduplicate by key

Several records share a business key, but one selected record should remain.

Preserve row order

Downstream processing depends on the original encounter order.

Allow arbitrary retention

No particular duplicate representative matters and execution can favor optimization.

Interview questions

QWhat does DataFrame.unique return, and what determines uniqueness?

It returns a DataFrame with duplicate rows removed. If subset is omitted, all columns participate in determining whether rows are duplicates.

Follow-up: How would you deduplicate records by a key while preserving the earliest input record? Use subset for the key, keep="first", and maintain_order=True.

QExplain the keep strategies.

keep accepts first, last, any, or none. first and last select a representative duplicate row, none removes every duplicated row, and any allows the optimizer to choose without guaranteeing which row remains.

Follow-up: Why might any be preferable? It permits more optimizations when the identity of the retained duplicate is irrelevant.

QHow can uniqueness be restricted or computed from selected columns?

subset accepts column names, selectors, or expressions and controls which values identify duplicates. The returned rows still contain the complete original schema, not only the subset columns.

Follow-up: How can uniqueness be based on a transformed value? Pass an expression such as pl.col("label").str.extract(...) in subset.

  • polars.DataFrame.is_unique
  • polars.DataFrame.n_unique
  • polars.Expr.unique

Source: polars.DataFrame.unique — Polars documentation. Checked 2026-10-15.