Remove exact duplicate rows
Preparing records for analysis or export
Pandas · Module
df.drop_duplicates
Remove duplicate rows from a DataFrame, optionally judging duplicates by selected columns. Choose whether to keep the first, last, or none of each duplicate group, and optionally reset the result index.
How it works
The method identifies repeated rows using every column unless subset selects particular columns. It returns a deduplicated DataFrame by default; keep chooses the retained occurrence, ignore_index creates a fresh sequential index, and inplace changes the original object instead of returning a DataFrame.
Syntax
DataFrame.drop_duplicates(subset=None, *, keep="first", inplace=False, ignore_index=False)Parameters
| Name | Type | Description |
|---|---|---|
subset | column label or iterable of labels, optional | Column label or iterable of labels used for identifying duplicates; if omitted, all columns are used. |
keep | {"first", "last", False}, default "first" | Which duplicate occurrence to retain: "first", "last", or False. |
inplace | bool, default False | If True, modify the existing DataFrame instead of returning a new one. |
ignore_index | bool, default False | If True, use a new default integer index in the result. |
Returns
DataFrame or None: a DataFrame with duplicates removed, or None when inplace=True.
Key points
- The method removes repeated rows and produces a deduplicated DataFrame by default.
- Pass subset to define which columns participate in duplicate detection.
- If subset is omitted, all columns are compared.
- keep selects the representative row from each duplicate group.
- keep="first" preserves the earliest occurrence in the current row order.
- keep="last" preserves the final occurrence in the current row order.
- keep=False removes every row that has a duplicate.
Common mistakes
- Duplicate detection does not use the DataFrame index, including a time-based index.
- With inplace=True, the operation mutates the DataFrame and returns None rather than a result DataFrame.
Example: Remove repeated records
import pandas as pd
df = pd.DataFrame({
"user": ["a", "a", "b"],
"score": [10, 10, 8],
})
unique = df.drop_duplicates(subset=["user"], keep="first", ignore_index=True)
print(unique)Keeps the first row for each user, considering only the user column, and rebuilds the result index.
When to use it
Deduplicate by key columns
Selecting one record per identifier
Discard all members of duplicate groups
Eliminating every ambiguous repeated record
Interview questions
QHow does drop_duplicates decide which rows are duplicates?
By default, duplicate detection considers all columns and retains the first occurrence. Use subset to restrict the comparison to selected columns.
Follow-up: How would you retain the latest record per key? Use subset for the key columns and keep="last", assuming the rows are already ordered by recency.
QWhat does the keep parameter control?
keep="first" retains the first duplicate occurrence, keep="last" retains the last, and keep=False removes every row belonging to a duplicate group.
Follow-up: What happens when keep=False is used on a group appearing three times? All three rows are removed.
QWhat are the return behavior and index options?
The method returns a DataFrame containing duplicates removed, unless inplace=True, in which case it returns None. ignore_index=True makes the returned rows use a new default index.
Follow-up: Why can relying on the old index be misleading after filtering? Removed rows leave gaps unless the index is explicitly rebuilt.
Related
DataFrame.duplicatedDataFrame.dropnaDataFrame.rename