Find repeated complete records
Data quality checks
Pandas · Module
df.duplicated
Returns a boolean Series identifying duplicate DataFrame rows. Compare all columns by default, or specify subset columns and choose whether to preserve the first, last, or none of each duplicate group.
How it works
Call this method to obtain a boolean Series aligned with the DataFrame index. Duplicate status is determined across all columns unless subset narrows the comparison, and keep controls which occurrence remains unmarked.
Syntax
DataFrame.duplicated(*subset=None*, *keep='first'*)Parameters
| Name | Type | Description |
|---|---|---|
subset | label or iterable of labels, optional | Column label or iterable of labels used for duplicate detection; if omitted, all columns are considered. |
keep | {'first', 'last', False}, optional | Controls which duplicate occurrences are marked: 'first', 'last', or False. Defaults to 'first'. |
Returns
A boolean Series indicating whether each row is duplicated according to keep.
Key points
- The method produces a boolean Series that identifies duplicate rows without modifying the DataFrame.
- A row is duplicated when the compared column values recur in another row.
- You can compare complete rows or restrict detection to selected columns.
- subset selects the columns used for comparison; omitting it uses every column.
- keep controls which occurrence of each duplicate group is marked True.
Common mistakes
- The default keep='first' leaves the first row in each duplicate group unmarked and flags later matches.
- With keep='last', the final row in each duplicate group is unmarked instead of the first.
- Using keep=False flags every row belonging to a duplicated group, including its first and last occurrences.
Example: Detect duplicate rows
import pandas as pd
df = pd.DataFrame({
"brand": ["A", "A", "B", "B"],
"model": ["x", "x", "y", "z"],
})
print(df.duplicated())
print(df.duplicated(subset=["brand"], keep=False))The first call checks complete rows, while the second checks only brand values and marks every member of each duplicate group.
When to use it
Detect repeated values across selected identifier columns
Business-key validation
Choose first or last occurrences before filtering duplicates
Record retention workflows
Interview questions
QWhat does DataFrame.duplicated return?
It returns a boolean Series with one value per DataFrame row. A True value means that row is considered duplicated under the selected subset and keep policy.
Follow-up: How would you identify all rows involved in duplication? Use df.duplicated(keep=False), which marks every occurrence in each duplicated group.
QHow does subset change duplicate detection?
If subset is omitted, all columns participate in duplicate detection. If supplied, it can be one column label or an iterable of labels, so only those columns determine equality.
Follow-up: How can you detect duplicate business keys while retaining other columns? Pass the key columns through subset, then use the resulting mask to select or inspect full rows.
QExplain the keep parameter.
keep accepts 'first', 'last', or False and defaults to 'first'. The first option flags all but the first occurrence, the second flags all but the last, and False flags every occurrence.
Follow-up: When is keep='last' useful? It is useful when the final record should be retained as the representative and earlier matching records should be identified.
Related
DataFrame.drop_duplicatesDataFrame.equalsSeries.duplicated