Lesson 10 of 13 · 12 min

Overfitting, Regularization, and Cross-Validation

A model overfits when it memorizes noise specific to your training rows instead of the biological signal that generalizes. This is the default failure mode on small omics datasets, where the feature count often dwarfs the sample count and rows are rarely independent. When replicates, cells, or repeated measurements come from the same patient or batch, a random split leaks near-duplicate rows across train and test, so you must split by group.

Diagnosing With Learning Curves

python

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import learning_curve, GroupKFold

# X: (n_samples, n_features) ndarray, y: binary label ndarray, groups: ndarray of patient IDs
pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression(C=10.0, max_iter=5000)),
])

sizes, train_scores, val_scores = learning_curve(
    pipe, X, y, groups=groups,
    cv=GroupKFold(n_splits=5),
    train_sizes=np.linspace(0.2, 1.0, 5),
    scoring="roc_auc",
)

for n, tr, va in zip(sizes, train_scores.mean(1), val_scores.mean(1)):
    print(f"n={n:<4d} train AUC={tr:.3f}  val AUC={va:.3f}  gap={tr-va:.3f}")

n=38   train AUC=1.000  val AUC=0.672  gap=0.328
n=76   train AUC=0.998  val AUC=0.719  gap=0.279
n=115  train AUC=0.994  val AUC=0.751  gap=0.243
n=153  train AUC=0.991  val AUC=0.773  gap=0.218
n=192  train AUC=0.988  val AUC=0.786  gap=0.202

A near-perfect train AUC with a much lower validation AUC, where the gap closes only slowly as you add data, is the signature of high variance. You reduce it by shrinking model capacity. In LogisticRegression and SVC, C is the inverse regularization strength, so a smaller C applies a stronger L2 penalty; in Ridge and Lasso, alpha is the direct strength, so a larger alpha penalizes more. For tree and forest models, lower max_depth or raise min_samples_leaf for the same effect.

Regularize And Cross-Validate

python

from sklearn.model_selection import cross_val_score

cv = GroupKFold(n_splits=5)
for C in [0.01, 0.1, 1.0, 10.0]:
    pipe = Pipeline([
        ("scaler", StandardScaler()),
        ("clf", LogisticRegression(penalty="l2", C=C, max_iter=5000)),
    ])
    auc = cross_val_score(pipe, X, y, groups=groups, cv=cv, scoring="roc_auc")
    print(f"C={C:<5} AUC = {auc.mean():.3f} +/- {auc.std():.3f}")

C=0.01  AUC = 0.781 +/- 0.041
C=0.1   AUC = 0.842 +/- 0.036
C=1.0   AUC = 0.826 +/- 0.048
C=10.0  AUC = 0.787 +/- 0.061

Warning: every data-dependent transform (StandardScaler, SelectKBest, PCA, imputation) must live inside the Pipeline so it is refit on the training fold alone. Fitting a scaler or feature selector on the full matrix before cross-validation leaks test statistics and silently inflates your score. Pair the Pipeline with GroupKFold so no patient, cell line, or sequencing batch appears on both sides of a split.

Nested CV For Honest Estimates

python

from sklearn.model_selection import GridSearchCV

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression(penalty="l2", max_iter=5000)),
])
param_grid = {"clf__C": [0.01, 0.1, 1.0, 10.0]}

outer = GroupKFold(n_splits=5)
scores = []
for train, test in outer.split(X, y, groups):
    inner = GroupKFold(n_splits=4)
    search = GridSearchCV(pipe, param_grid, cv=inner, scoring="roc_auc")
    # GridSearchCV.fit forwards groups to the inner GroupKFold splitter
    search.fit(X[train], y[train], groups=groups[train])
    scores.append(search.score(X[test], y[test]))

print(f"nested CV AUC = {np.mean(scores):.3f} +/- {np.std(scores):.3f}")
# honest estimate, typically below the best tuned score above (e.g. ~0.821)

Try it yourself: swap LogisticRegression for RandomForestClassifier and move max_depth and min_samples_leaf into the GridSearchCV param_grid (as clf__max_depth and clf__min_samples_leaf). Rerun the nested loop and compare the honest nested AUC against the single best tuned cross_val_score from earlier. The nested estimate should come out lower, because that difference is the optimism you removed by refusing to select hyperparameters on the same data you report.

Evaluation Metrics for Imbalanced Biology Deep Learning Teaser: CNNs on Sequence