Fallback data completion
Populate incomplete records from a secondary DataFrame while preserving values already present in the primary frame.
Pandas · Module
df.combine_first
Fills missing values in one DataFrame from corresponding non-missing values in another. The result uses the union of both DataFrames’ row and column indexes, while the first DataFrame takes precedence where both contain values.
How it works
Call combine_first with another DataFrame to create a new DataFrame that keeps the caller's non-missing cells and uses the other frame only for missing cells at corresponding row and column labels. The output expands to the union of both indexes and columns; a missing value remains where the other frame has no matching location or is also missing.
Syntax
DataFrame.combine_first(other)Parameters
| Name | Type | Description |
|---|---|---|
other | DataFrame | DataFrame used to supply values for missing cells. |
Returns
DataFrame containing the combined values from the two DataFrames.
Key points
- The method targets missing cells and fills them from the corresponding location in the other DataFrame.
- It combines two DataFrames while using the second frame as a source for missing values.
- Both row labels and column labels in the output cover the union of the inputs.
- The calling DataFrame supplies the preferred values wherever it already has data.
- When both aligned cells are populated, the first DataFrame wins rather than the fallback.
Common mistakes
- A missing value is not filled when the other DataFrame has no corresponding row-and-column location.
Example: Fill missing cells from a fallback DataFrame
import pandas as pd
primary = pd.DataFrame({"score": [10, None], "name": ["A", None]}, index=[1, 2])
fallback = pd.DataFrame({"score": [20, 30], "name": ["B", "C"]}, index=[2, 3])
result = primary.combine_first(fallback)
print(result)Missing cells in primary are filled from fallback at matching labels; the result also includes labels found only in either input.
When to use it
Label-aligned consolidation
Merge frames with partially different row or column labels while retaining all labels.
Priority-based data reconciliation
Prefer a curated or current dataset and use an older or secondary dataset only for gaps.
Interview questions
QHow does combine_first resolve overlapping values?
It fills missing values in the calling DataFrame with non-missing values from the other DataFrame at the same labels. Values already present in the caller take precedence when both frames contain non-missing values.
Follow-up: What happens to the shape? The result uses the union of the two row indexes and column indexes, so it can contain rows or columns introduced by the other frame.
QWhat does the method accept and return?
The argument is another DataFrame. It supplies candidate values for null cells in the caller, and the operation returns a DataFrame rather than modifying the caller in place.
Follow-up: What if the fallback has a value under a new label? That label is included in the returned frame because the indexes are combined by union.
QWhy might nulls remain after calling combine_first?
Alignment is label-based: a fallback value is used only when its row and column labels correspond to a missing cell in the first DataFrame. Non-missing values in the first frame remain authoritative.
Follow-up: Can it fill every null? No. Missingness remains when the fallback lacks the location or also provides a missing value.
Related
DataFrame.combineDataFrame.updateDataFrame.fillna