Random Forests and Gradient Boosting

Most biological prediction tasks arrive as a tabular feature matrix: samples in rows, engineered features such as expression values, k-mer counts, or physicochemical descriptors in columns. On this shape, tree ensembles are the strongest low-tuning default we have, usually beating linear models and matching neural networks with a fraction of the effort. They handle mixed scales, nonlinear interactions, and missing values with almost no preprocessing.

Decision Trees

A decision tree learns greedy, recursive binary splits that minimize node impurity, typically the Gini index or entropy for classification. A single deep tree can fit the training set almost perfectly, but it has high variance: a small perturbation of the data changes which feature is chosen at the top and cascades into a different structure. That instability is exactly what ensembles exploit and average away.

Bagging and Random Forests

A random forest trains many trees on bootstrap resamples of the data and restricts each split to a random subset of features via max_features. Bootstrapping plus feature subsampling decorrelates the trees, so averaging their votes cuts variance with little added bias. Samples left out of a given bootstrap draw, the out-of-bag rows, provide a built-in validation estimate for free.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
import numpy as np

X, y = load_breast_cancer(return_X_y=True, as_frame=True)

rf = RandomForestClassifier(
    n_estimators=400,
    max_features="sqrt",
    oob_score=True,
    n_jobs=-1,
    random_state=0,
)

scores = cross_val_score(rf, X, y, cv=5, scoring="roc_auc")
print(f"5-fold ROC AUC: {scores.mean():.3f} +/- {scores.std():.3f}")

rf.fit(X, y)
print(f"OOB score: {rf.oob_score_:.3f}")

top = np.argsort(rf.feature_importances_)[::-1][:5]
for i in top:
    print(f"{X.columns[i]:<24} {rf.feature_importances_[i]:.3f}")

5-fold ROC AUC: 0.992 +/- 0.006
OOB score: 0.963
worst perimeter          0.129
worst concave points     0.125
worst area               0.113
worst radius             0.112
mean concave points      0.104

Boosting Ensembles

Boosting builds trees sequentially, where each shallow tree corrects the residual errors of the ensemble so far rather than voting independently. HistGradientBoostingClassifier bins each feature into integer histograms, which makes training fast on wide biological matrices and gives native handling of NaN values. XGBoost adds explicit regularization and column subsampling and remains a strong, well-defended tabular baseline.

from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
from xgboost import XGBClassifier

X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)

hgb = HistGradientBoostingClassifier(
    learning_rate=0.1, max_iter=300, early_stopping=True, random_state=0
)
hgb.fit(X_tr, y_tr)

xgb = XGBClassifier(
    n_estimators=400, learning_rate=0.05, max_depth=3,
    subsample=0.8, colsample_bytree=0.8,
    tree_method="hist", eval_metric="logloss", random_state=0,
)
xgb.fit(X_tr, y_tr)

for name, m in [("HistGB", hgb), ("XGB", xgb)]:
    auc = roc_auc_score(y_te, m.predict_proba(X_te)[:, 1])
    print(f"{name:<7} test ROC AUC: {auc:.3f}")

perm = permutation_importance(
    hgb, X_te, y_te, n_repeats=20, scoring="roc_auc", random_state=0
)
for i in perm.importances_mean.argsort()[::-1][:5]:
    print(f"{X.columns[i]:<24} {perm.importances_mean[i]:.3f}")