General regression
Predict continuous numeric targets from tabular features.
Scikit-Learn · Class
GradientBoostingRegressor
GradientBoostingRegressor fits an additive regression model by sequentially training trees against the negative gradient of a chosen loss. Tune learning_rate and n_estimators together to balance each tree’s contribution against the number of stages.
How it works
The estimator builds an additive regression model stage by stage. At each stage, it fits a regression tree to the negative gradient of the selected differentiable loss, while learning_rate controls each tree’s contribution.
Syntax
class sklearn.ensemble.GradientBoostingRegressor(*, loss='squared_error', learning_rate=0.1, n_estimators=100, subsample=1.0, criterion='deprecated', min_samples_split=2, min_samples_leaf=1, max_depth=3, min_weight_fraction_leaf=0.0, max_features=None, max_leaf_nodes=None, min_impurity_decrease=0.0, init=None, random_state=None, max_features=None, alpha=0.9, verbose=0, max_leaf_nodes=None, warm_start=False, validation_fraction=0.1, n_iter_no_change=None, tol=0.0001, ccp_alpha=0.0)Parameters
| Name | Type | Description |
|---|---|---|
loss | {'squared_error', 'absolute_error', 'huber', 'quantile'}, default='squared_error' | Loss function to optimize: 'squared_error', 'absolute_error', 'huber', or 'quantile'. |
learning_rate | float, default=0.1 | Shrinks each tree’s contribution. |
n_estimators | int, default=100 | Number of boosting stages. |
subsample | float, default=1.0 | Fraction of samples used to fit each base learner. |
criterion | deprecated | Split-quality criterion; the current documentation marks this parameter deprecated. |
min_samples_split | int or float, default=2 | Minimum samples required to split an internal node. |
min_samples_leaf | int or float, default=1 | Minimum samples required at a leaf. |
max_depth | int, default=3 | Maximum tree depth. |
init | estimator or 'zero', default=None | Initial estimator used to compute the starting predictions. |
random_state | int, RandomState instance, or None, default=None | Controls pseudo-randomness. |
max_features | int, float, {'sqrt', 'log2'}, or None, default=None | Number of features considered for the best split. |
alpha | float, default=0.9 | Quantile used by the quantile loss. |
Returns
fit returns the fitted estimator; predict returns regression predictions; score returns the coefficient of determination R²; apply returns leaf indices.
Key points
- Use this class for supervised regression with gradient-boosted decision trees.
- Predictions are formed by adding the contributions learned at successive boosting stages.
- Each new tree follows the negative gradient of the currently selected loss.
- The default objective is squared error, with additional absolute_error, huber, and quantile choices.
- A smaller learning rate makes each individual tree contribute less to the ensemble.
- n_estimators determines how many sequential trees the model trains.
- subsample controls how much of the training sample each tree sees.
Common mistakes
- Changing learning_rate should be considered together with n_estimators; reducing each tree’s contribution generally calls for more boosting stages.
- For intermediate and large datasets, consider HistGradientBoostingRegressor instead because it is documented as much faster.
- When loss is quantile, set alpha to the desired quantile.
Example: Train and predict
from sklearn.datasets import make_regression
from sklearn.ensemble import GradientBoostingRegressor
X, y = make_regression(n_samples=200, n_features=4, noise=0.2, random_state=0)
model = GradientBoostingRegressor(n_estimators=100, learning_rate=0.05, random_state=0)
model.fit(X, y)
predictions = model.predict(X[:5])
print(predictions)Creates regression data, trains the estimator, and predicts five targets.
When to use it
Quantile regression
Estimate conditional quantiles rather than only a conditional mean.
Robust regression
Reduce sensitivity to unusual target values.
Nonlinear tabular prediction
Model nonlinear relationships and feature interactions with tree ensembles.
Interview questions
QHow does GradientBoostingRegressor work?
It is a regression ensemble that builds an additive model in forward stage-wise fashion. Each stage fits a regression tree to the negative gradient of the chosen loss.
Follow-up: How can model capacity be adjusted? Use learning_rate and n_estimators together, along with tree and sampling parameters such as max_depth and subsample.
QWhat does the loss parameter control?
loss selects the objective: squared_error, absolute_error, huber, or quantile. The default is squared_error; quantile regression uses alpha to select the quantile.
Follow-up: Which loss is robust to outliers? absolute_error is documented as a robust regression loss, while huber combines squared and absolute error.
QWhat do n_estimators and subsample mean?
n_estimators is the number of boosting stages, and subsample is the fraction of samples used to fit each base learner. A subsample below 1.0 creates stochastic gradient boosting.
Follow-up: What is the practical alternative for large datasets? HistGradientBoostingRegressor is documented as much faster for n_samples at least 10,000.
Related
HistGradientBoostingRegressorGradientBoostingClassifierRandomForestRegressorRandomForestClassifierMultiOutputRegressor