Evaluation Metrics for Imbalanced Biology

In many biological classification tasks the positive class is vanishingly rare: pathogenic variants are a fraction of a percent of all observed variants, and disease cases are often swamped by controls. Throughout this lesson we fix the convention that the positive label 1 means pathogenic, the minority class we actually care about. Choosing the wrong evaluation metric on data like this will make a useless model look excellent.

The Confusion Matrix

The confusion matrix counts four outcomes: true negatives TN, false positives FP, false negatives FN, and true positives TP. The function sklearn.metrics.confusion_matrix returns them ordered as [[TN, FP], [FN, TP]] when the labels are [0, 1]. Precision is TP divided by TP plus FP, the fraction of positive calls that are real; recall, also called sensitivity, is TP divided by TP plus FN, the fraction of real positives you caught; F1 is the harmonic mean of precision and recall; accuracy is TP plus TN over all samples.

import numpy as np
from sklearn.metrics import (confusion_matrix, accuracy_score,
                             precision_score, recall_score, f1_score)

# 1000 variants, 1 percent truly pathogenic (positive class = 1)
rng = np.random.default_rng(0)
y_true = np.zeros(1000, dtype=int)
y_true[:10] = 1                       # 10 pathogenic, 990 benign
rng.shuffle(y_true)

# A lazy model that calls every variant benign
y_pred = np.zeros_like(y_true)

print("confusion matrix [[TN, FP], [FN, TP]]:")
print(confusion_matrix(y_true, y_pred))
print(f"accuracy : {accuracy_score(y_true, y_pred):.3f}")
print(f"precision: {precision_score(y_true, y_pred, zero_division=0):.3f}")
print(f"recall   : {recall_score(y_true, y_pred, zero_division=0):.3f}")
print(f"f1       : {f1_score(y_true, y_pred, zero_division=0):.3f}")

confusion matrix [[TN, FP], [FN, TP]]:
[[990   0]
 [ 10   0]]
accuracy : 0.990
precision: 0.000
recall   : 0.000
f1       : 0.000

Ranking Metrics And Thresholds

ROC-AUC and PR-AUC are threshold-free ranking metrics computed from predicted probabilities rather than hard labels. The function roc_auc_score integrates true-positive rate against false-positive rate; because false-positive rate is FP over FP plus TN and TN is enormous when positives are rare, ROC-AUC stays optimistic even when most positive calls are wrong. The function average_precision_score is the recommended PR-AUC, the area under the precision-recall curve, and its no-skill baseline equals the positive prevalence, so it exposes the real difficulty. The setting class_weight='balanced' reweights the loss inversely to class frequency, which mostly relocates the default 0.5 decision boundary and barely changes the score ranking.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (recall_score, precision_score, roc_auc_score,
                             average_precision_score, precision_recall_curve,
                             confusion_matrix)

X, y = make_classification(
    n_samples=20000, n_features=30, n_informative=12, n_redundant=4,
    weights=[0.98], class_sep=1.8, flip_y=0.0, random_state=42,
)
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=42)
print(f"test-set prevalence: {y_te.mean():.4f}")

# Ranking metrics are threshold-free; class_weight only moves the 0.5 operating point.
for cw in (None, "balanced"):
    clf = LogisticRegression(max_iter=2000, class_weight=cw).fit(X_tr, y_tr)
    proba = clf.predict_proba(X_te)[:, 1]
    pred = clf.predict(X_te)
    print(f"class_weight={str(cw):>8}  "
          f"[email protected]={recall_score(y_te, pred, zero_division=0):.2f}  "
          f"[email protected]={precision_score(y_te, pred, zero_division=0):.2f}  "
          f"ROC-AUC={roc_auc_score(y_te, proba):.3f}  "
          f"PR-AUC={average_precision_score(y_te, proba):.3f}")

# Choose a threshold by cost: a missed pathogenic call (FN) is unacceptable,
# so require recall >= 0.90 and take the most precise threshold that meets it.
clf = LogisticRegression(max_iter=2000).fit(X_tr, y_tr)
scores = clf.predict_proba(X_te)[:, 1]
prec, rec, thr = precision_recall_curve(y_te, scores)
prec, rec = prec[:-1], rec[:-1]           # align arrays with thresholds
keep = rec >= 0.90
i = np.argmax(prec[keep])
t = thr[keep][i]
print(f"chosen threshold={t:.3f}  recall={rec[keep][i]:.3f}  precision={prec[keep][i]:.3f}")
print(confusion_matrix(y_te, (scores >= t).astype(int)))

test-set prevalence: 0.0200
class_weight=    None  [email protected]=0.57  [email protected]=0.86  ROC-AUC=0.970  PR-AUC=0.771
class_weight=balanced  [email protected]=0.93  [email protected]=0.28  ROC-AUC=0.981  PR-AUC=0.745
chosen threshold=0.013  recall=0.900  precision=0.196
[[5437  443]
 [  12  108]]

The ROC-AUC of 0.970 looks superb, but the PR-AUC of 0.771 against a 0.02 prevalence baseline is the honest summary. Setting class_weight='balanced' lifted recall at the default threshold from 0.57 to 0.93 while ROC-AUC and PR-AUC barely moved, confirming it repositions the operating point rather than improving the ranking. At the cost-driven threshold of 0.013 the model reaches 0.90 recall at 0.196 precision, meaning 443 false alarms and 108 true calls but only 12 missed variants. Whether that trade is acceptable is a biological decision: in clinical variant triage a false negative is usually far costlier than a false positive a curator later filters out, so you optimize recall and PR-AUC, not accuracy.