Lesson 13 of 13 · 11 min

Pitfalls, Reproducibility, and Where to Next

A model that scores 0.95 AUROC in your notebook and 0.60 in the wet lab did not get unlucky; it was almost always measuring the wrong thing from the start. The gap between a real advance and an artifact is a short list of mistakes that recur across nearly every biological ML project. Treat what follows as a checklist you run before, not after, you decide to trust a number.

Data leakage is the deadliest: information from the test set reaches training, usually because homologous sequences or duplicate PDB entries land on both sides of a random split. Unrealistic baselines let a deep net look impressive when it was never compared against BLAST nearest-neighbor, a PSSM, or k-mer logistic regression, and metric mismatch reports accuracy on a 5 percent-positive set where a constant predictor already hits 0.95 instead of AUPRC, MCC, or Spearman. Non-reproducible splits regenerate train and test membership on every run, so no two experiments are comparable.

Warning: a uniform random split leaks through evolutionary homology. Cluster sequences first with MMseqs2 (mmseqs easy-cluster seqs.fasta clust tmp --min-seq-id 0.3 -c 0.8), then assign whole clusters to a single fold so no test protein shares more than 30 percent identity with any training protein.

A Pre-Flight Checklist

text

[ ] Splits are homology-aware (cluster by identity), not uniform-random
[ ] Test set frozen and content-hashed before any model is trained
[ ] At least one non-deep baseline reported: BLAST / PSSM / k-mer LR
[ ] Metric matches class balance: AUPRC or MCC for imbalance, Spearman for regression
[ ] No target-derived features (labels, near-identical homologs) in the inputs
[ ] Preprocessing (scaling, vocab) fit on train only, then applied to test
[ ] All RNG seeds fixed; run repeated to report mean +/- std over seeds
[ ] Data, splits, code commit, and environment all pinned to versions

Seeds And Versioned Data

python

import os, random
import numpy as np
import torch

def seed_everything(seed: int = 42) -> None:
    os.environ["PYTHONHASHSEED"] = str(seed)
    # Required for deterministic cuBLAS ops on CUDA >= 10.2
    os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    # Force deterministic kernels; raises if an op lacks a deterministic impl
    torch.use_deterministic_algorithms(True)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

seed_everything(42)

bash

# Track the frozen dataset and split files with DVC
dvc add data/raw/sequences.fasta data/splits/
# DVC writes ignore rules into per-directory .gitignore files, so stage those
git add data/raw/sequences.fasta.dvc data/splits.dvc data/raw/.gitignore data/.gitignore

# Record the exact clustering + split parameters in the commit
git commit -m "freeze splits: mmseqs id=0.3 cov=0.8, cluster-based 80/10/10"
git tag -a data-v1.0 -m "clustered split, seed=42"

# Anyone can now restore the identical bytes:
#   git checkout data-v1.0 && dvc checkout

Model Cards

python

# model_card.py -- ships beside the checkpoint; dump with yaml.safe_dump
MODEL_CARD = {
    "model": "esm2_t33_finetuned_thermostability",
    "version": "1.2.0",
    "date": "2026-07-13",
    "intended_use": "Predict melting temperature (Tm) of single-chain proteins < 400 aa",
    "out_of_scope": ["multi-chain complexes", "membrane proteins", "non-natural residues"],
    "training_data": {
        "source": "Meltome Atlas (dvc tag data-v1.0)",
        "split": "MMseqs2 cluster-based, 30% identity, 80/10/10",
        "n_train": 34211,
    },
    "metrics": {
        "test_spearman": 0.71,
        "test_mae_celsius": 4.8,
        "baseline_pssm_spearman": 0.52,
    },
    "reproducibility": {
        "seed": 42,
        "code_commit": "a1b9f3c",
        "framework": ["torch==2.4.1", "fair-esm==2.0.0"],
    },
    "ethical_considerations": "Not validated for clinical or biosafety decisions",
}

Once your evaluation is honest, the frontier is representation. Protein language models such as ESM-2 (esm2_t33_650M_UR50D) give per-residue embeddings that beat handcrafted features on most function and stability tasks, while graph neural networks over residue-contact or atomic graphs (via PyTorch Geometric) capture 3D structure that sequence-only models miss. Validate any new idea on community benchmarks with fixed splits, TAPE for sequence tasks and ProteinGym for variant-effect prediction, so your numbers are comparable to published baselines rather than only to your own.

Try it yourself: take a dataset you have already benchmarked, re-split it with mmseqs easy-cluster at 30 percent identity, and retrain your best model. If Spearman or AUPRC drops sharply, the earlier score was inflated by homology leakage; write both numbers into a model card and keep the honest one.

Applications: Variant Effect, Protein Function, Expression