Classify two-category records with staged tree corrections.
Tabular binary classification
Scikit-Learn · Class
GradientBoostingClassifier
GradientBoostingClassifier builds a classification model by adding regression trees stage by stage to optimize a differentiable loss. It supports binary and multiclass classification, with log loss as the default objective.
How it works
The classifier constructs an additive prediction model incrementally. At each stage, regression trees are fitted to the loss gradient, and their contributions are scaled before being added to the ensemble.
Syntax
class sklearn.ensemble.GradientBoostingClassifier(loss='log_loss', learning_rate=0.1, n_estimators=100, subsample=1.0, criterion='friedman_mse', min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_depth=3, min_impurity_decrease=0.0, init=None, random_state=None, max_features=None, verbose=0, max_leaf_nodes=None, warm_start=False, validation_fraction=0.1, n_iter_no_change=None, tol=0.0001, ccp_alpha=0.0)Parameters
| Name | Type | Description |
|---|---|---|
loss | {'log_loss', 'exponential'}, default='log_loss' | Loss function to optimize: 'log_loss' or 'exponential'. Default: 'log_loss'. |
learning_rate | float, default=0.1 | Shrinkage applied to each tree contribution. Default: 0.1. |
n_estimators | int, default=100 | Number of boosting stages. Default: 100. |
subsample | float, default=1.0 | Fraction of samples used for each base learner. Default: 1.0. |
min_samples_split | int or float, default=2 | Minimum number of samples required to split an internal node. Default: 2. |
min_samples_leaf | int or float, default=1 | Minimum number of samples required at a leaf. Default: 1. |
min_weight_fraction_leaf | float, default=0.0 | Minimum weighted fraction of samples required at a leaf. Default: 0.0. |
max_depth | int, default=3 | Maximum depth of each individual regression estimator. Default: 3. |
init | estimator or 'zero', default=None | Estimator providing the initial predictions. Default: None. |
random_state | int, RandomState instance or None, default=None | Controls randomness in subsampling and feature selection. |
max_features | int, float, {'sqrt', 'log2'} or None, default=None | Number of features considered for the best split. Default: None. |
verbose | int, default=0 | Verbosity level. Default: 0. |
Returns
An instance of GradientBoostingClassifier.
Key points
- The model grows sequentially: each new tree contributes an additional correction to the current additive prediction.
- Each boosting stage fits class-specific regression trees against the negative loss gradient.
- For two-class problems, each stage needs only one regression tree.
- The default log-loss option uses the same binomial or multinomial deviance family associated with logistic regression.
- learning_rate controls shrinkage: it scales down every tree's contribution before it is added.
- The loss parameter selects the objective optimized during boosting; stable documentation lists log_loss and exponential.
- subsample determines what fraction of training samples each base regression tree receives.
- Setting subsample below 1.0 activates stochastic gradient boosting rather than using every sample at every stage.
Common mistakes
- Reducing learning_rate generally requires increasing n_estimators to preserve model capacity; tune these two parameters together.
- Choosing exponential loss changes the procedure to the AdaBoost equivalent, so use it deliberately rather than as a cosmetic loss choice.
Example: Fit and predict probabilities
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier
X, y = make_classification(random_state=0)
model = GradientBoostingClassifier(random_state=0)
model.fit(X, y)
print(model.predict_proba(X[:2]))Creates a synthetic binary dataset, fits the classifier, and prints class probabilities for two samples.
When to use it
Model several classes using class-specific trees at each stage.
Tabular multiclass classification
Use subsample below 1.0 to fit each stage on a sample of the training data.
Stochastic boosting
Combine warm starts with estimator-count changes when incrementally extending a fitted ensemble.
Controlled model growth
Interview questions
QHow does GradientBoostingClassifier work?
It is a stage-wise additive classifier that fits regression trees to the negative gradient of the selected differentiable loss. Binary classification uses one regression tree per stage, while multiclass classification uses one for each class.
Follow-up: How would you reduce variance? Use subsampling below 1.0, accepting the documented trade-off of increased bias.
QWhat loss functions are supported?
The loss can be log loss or exponential loss, with log loss as the default. Log loss corresponds to binomial or multinomial deviance, while exponential loss recovers AdaBoost.
Follow-up: What is the practical consequence of changing loss? It changes the optimization objective and, for exponential loss, the resulting boosting procedure.
QWhat are the key capacity controls?
learning_rate shrinks each tree's contribution, while n_estimators controls how many boosting stages are fitted. They trade off against one another, so a smaller learning rate commonly calls for more estimators.
Follow-up: How can training continue after fitting? Set warm_start=True and add estimators to an already fitted model.
Related
GradientBoostingRegressorHistGradientBoostingClassifierAdaBoostClassifier