Ensemble a high-variance classifier
Reducing prediction variance
Scikit-Learn · Class
BaggingClassifier
An ensemble classifier that trains base classifiers on randomized subsets of samples and features, then aggregates their predictions. It is commonly used to reduce variance in high-variance estimators.
How it works
BaggingClassifier trains multiple copies of a base classifier on randomized subsets of samples and, optionally, features. It combines their outputs to produce a classification result, commonly reducing the variance of a high-variance estimator such as a decision tree.
Syntax
class sklearn.ensemble.BaggingClassifier(*estimator=None*, *n_estimators=10*, ***, *max_samples=None*, *max_features=1.0*, *bootstrap=True*, *bootstrap_features=False*, *oob_score=False*, *warm_start=False*, *n_jobs=None*, *random_state=None*, *verbose=0*)Parameters
| Name | Type | Description |
|---|---|---|
estimator | object, default=None | Base estimator fitted on random subsets; defaults to DecisionTreeClassifier when None. |
n_estimators | int, default=10 | Number of base estimators in the ensemble. |
max_samples | int or float, default=None | Number or fraction of samples drawn for each estimator. |
max_features | int or float, default=1.0 | Number or fraction of features drawn for each estimator. |
bootstrap | bool, default=True | Whether samples are drawn with replacement. |
bootstrap_features | bool, default=False | Whether features are drawn with replacement. |
oob_score | bool, default=False | Whether to estimate out-of-bag performance. |
warm_start | bool, default=False | Whether to reuse an existing fitted ensemble when adding estimators. |
n_jobs | int, default=None | Number of parallel jobs; None means one job unless configured otherwise. |
random_state | int, RandomState instance or None | Controls randomness used by sampling. |
verbose | int, default=0 | Verbosity level. |
Returns
A fitted BaggingClassifier is returned by fit; prediction methods return classes, class probabilities, log-probabilities, or averaged decision scores as applicable.
Key points
- The class combines multiple fitted classifiers into one ensemble predictor.
- Each member is trained using a randomized subset of the original training data.
- The ensemble combines member outputs rather than relying on a single classifier.
- Classification outputs can be combined through voting or averaging, depending on the prediction method.
- max_features controls how many input features are selected for each base estimator.
- max_samples controls the number or fraction of observations assigned to each member.
- decision_function returns the average decision output from the ensemble's base classifiers.
Common mistakes
- Leaving estimator as None uses a DecisionTreeClassifier rather than a custom classifier.
- bootstrap changes whether each training subset samples observations with replacement.
- The supplied estimator must be suitable for fitting on each randomized subset.
Example: Train and predict with bagging
from sklearn.datasets import make_classification
from sklearn.ensemble import BaggingClassifier
from sklearn.svm import SVC
X, y = make_classification(n_samples=100, n_features=4, random_state=0)
model = BaggingClassifier(
estimator=SVC(probability=True),
n_estimators=10,
random_state=0,
)
model.fit(X, y)
print(model.predict([[0, 0, 0, 0]]))This trains an ensemble of SVC classifiers on randomized subsets and predicts a class for one sample.
When to use it
Create diverse base models
Randomized feature and sample selection
Build multiple classifiers concurrently
Parallel ensemble training
Interview questions
QWhat problem does BaggingClassifier address?
BaggingClassifier is an ensemble meta-estimator. It fits base classifiers on random subsets of the training data and aggregates their predictions, which can reduce variance.
Follow-up: How do sample and feature subsampling differ? max_samples controls observations, while max_features controls input features.
QWhat does the estimator parameter control?
estimator selects the classifier used in every ensemble member. If it is None, scikit-learn uses a DecisionTreeClassifier.
Follow-up: What is a useful way to obtain probabilistic predictions? Use a base estimator that supports probability estimates, then call predict_proba.
QWhat does n_estimators mean?
n_estimators specifies how many base classifiers are built. Increasing it generally makes the ensemble larger and can stabilize aggregation, at additional computational cost.
Follow-up: How can training parallelism be configured? Set n_jobs to control the number of jobs used for parallel execution.
Related
BaggingRegressorAdaBoostClassifierVotingClassifierRandomForestClassifierDecisionTreeClassifier