Out-of-fold diagnostics
Inspect prediction errors without evaluating only in-sample outputs.
Scikit-Learn · Function
cross_val_predict
Generate one out-of-fold estimate for every input sample by fitting separate estimators on cross-validation training folds. Use it to obtain predictions for visualization or downstream analysis, not as a substitute for per-fold generalization metrics.
How it works
The function splits the data using the selected cross-validation strategy. For each split, it fits a fresh estimator on the training indices and applies the requested prediction method to the held-out indices, assembling one result for every input sample.
Syntax
sklearn.model_selection.cross_val_predict(estimator, X, y=None, *, groups=None, cv=None, n_jobs=None, verbose=0, params=None, pre_dispatch='2*n_jobs', method='predict')Parameters
| Name | Type | Description |
|---|---|---|
estimator | estimator | Estimator instance used to fit the data; it must provide fit and the method selected by method. |
X | array-like or sparse matrix | Input data with shape (n_samples, n_features). |
y | array-like or sparse matrix, default=None | Target values for supervised learning; defaults to None. |
groups | array-like, default=None | Group labels used by the splitter; defaults to None. |
cv | int, cross-validation generator, or iterable, default=None | Cross-validation strategy: None, an integer, a splitter, or an iterable of train/test index arrays. |
n_jobs | int, default=None | Number of parallel jobs; defaults to None. |
verbose | int, default=0 | Controls verbosity; defaults to 0. |
params | dict, default=None | Extra parameters passed to the estimator's fit and scoring calls. |
pre_dispatch | int or str, default='2*n_jobs' | Number of jobs dispatched during parallel execution; defaults to '2*n_jobs'. |
method | str, default='predict' | Estimator method used to generate predictions; defaults to 'predict'. |
Returns
predictions: ndarray or other estimator-method output, containing the result of calling method on each sample's held-out fold.
Key points
- Produces one cross-validated estimate for every input row.
- The cv argument determines how samples are divided into training and test portions.
- Each held-out prediction comes from a model trained without that sample.
- The estimator needs fit plus the prediction interface selected by method.
- Parallel work is dispatched in batches controlled by pre_dispatch, whose default is 2*n_jobs.
Common mistakes
- Do not treat one metric computed over the concatenated predictions as a reliable generalization estimate; use cross_validate or cross_val_score for per-fold evaluation.
- Every sample receives one held-out prediction, so this is not repeated prediction averaging across folds.
Example: Generate out-of-fold predictions
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_predict
X, y = load_diabetes(return_X_y=True)
predictions = cross_val_predict(Ridge(), X, y, cv=5)
print(predictions.shape)Fits one estimator per fold and returns an out-of-fold prediction for every sample.
When to use it
Leakage-aware downstream features
Create a prediction for every training sample while ensuring its model did not train on that sample.
Prediction-error plots
Visualize observed versus cross-validated predictions.
Interview questions
QHow does cross_val_predict work?
It partitions the data according to cv, fits an estimator on each training portion, and predicts the corresponding held-out portion. The outputs are arranged so each input sample has one cross-validated estimate.
Follow-up: Why can a single metric over these outputs be misleading? Fold sizes or distributions can differ, so aggregate predictions may not represent a valid generalization estimate; calculate metrics per fold instead.
QWhat does cv control?
cv selects the splitting strategy. None uses the default five-fold strategy, an integer specifies the number of folds, and a splitter or iterable can provide explicit train/test index pairs.
Follow-up: What should be passed when samples are grouped? Supply group labels through groups, or through params when metadata routing is enabled.
QWhat contract must estimator satisfy?
The estimator must implement fit and the method named by method. The default method is predict, while supported estimator methods can produce alternatives such as probabilities or decision scores.
Follow-up: What happens if an estimator lacks the requested method? The call cannot perform that prediction operation and raises an error.
Related
cross_validatecross_val_scoreKFoldStratifiedKFoldGroupKFold