Train/Val/Test Splits and Sequence-Similarity Leakage

A split exists to estimate how a model behaves on data it has never seen. The training set fits parameters, the validation set tunes hyperparameters and triggers early stopping, and the test set is touched once to report a final number. If any information about validation or test leaks into training, that number becomes optimistic and the model fails silently after deployment.

Why Random Splits Leak

Biological sequences are not independent draws. Proteins fall into families, and a typical UniProt export contains many near-identical orthologs, paralogs, and strain variants of the same underlying molecule. A uniform random split scatters members of one family across train, validation, and test, so the model can memorize a family during training and then be graded on its close relatives.

Cluster By Similarity

# Cluster at 30% identity, 80% target coverage (catches remote homology)
mmseqs easy-cluster sequences.fasta clusterRes tmp \
  --min-seq-id 0.3 -c 0.8 --cov-mode 1

# CD-HIT alternative; it cannot cluster below 40% identity, and -n tracks -c
# cd-hit -i sequences.fasta -o cdhit40 -c 0.4 -n 2 -M 0 -T 0

Create sequence database
Number of sequences: 12483

Clustering
Time for clustering: 0h 0m 4s 118ms

Writing cluster results
clusterRes_cluster.tsv  clusterRes_rep_seq.fasta  clusterRes_all_seqs.fasta
Number of clusters: 3164

import pandas as pd
from sklearn.model_selection import GroupShuffleSplit, GroupKFold

df = pd.read_csv("labels.csv")                 # columns: id, y
edges = pd.read_csv("clusterRes_cluster.tsv", sep="\t",
                    names=["rep", "member"])   # representative <tab> member
df["cluster"] = df["id"].map(dict(zip(edges.member, edges.rep)))
groups = df["cluster"].to_numpy()

# Carve out a homology-disjoint test set (~15% of clusters), touched once.
gss = GroupShuffleSplit(n_splits=1, test_size=0.15, random_state=0)
dev_idx, test_idx = next(gss.split(df, df["y"], groups))

# Cross-validate on the remaining clusters; no cluster spans train and val.
gkf = GroupKFold(n_splits=5)
dev_groups = groups[dev_idx]
for k, (tr, va) in enumerate(gkf.split(dev_idx, df["y"].iloc[dev_idx], dev_groups)):
    shared = set(dev_groups[tr]) & set(dev_groups[va])
    print(f"fold {k}: train={len(tr)} val={len(va)} shared_clusters={len(shared)}")

fold 0: train=8493 val=2124 shared_clusters=0
fold 1: train=8481 val=2136 shared_clusters=0
fold 2: train=8500 val=2117 shared_clusters=0
fold 3: train=8487 val=2130 shared_clusters=0
fold 4: train=8507 val=2110 shared_clusters=0

Cluster-disjoint splits remove homology leakage but not every leak. Batch effects appear when positives and negatives come from different experiments or databases, so the model learns the assay signature instead of biology; verify that label balance and metadata sources are matched across folds. Target leakage appears when preprocessing sees the answer, for example when a scaler, embedding, or feature-selection step is fit on the whole dataset before splitting, so wrap every transform in a Pipeline that only sees training rows.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

# Label rate per fold: large swings flag batch effects or class imbalance.
for k, (tr, va) in enumerate(gkf.split(dev_idx, df["y"].iloc[dev_idx], dev_groups)):
    rate = df["y"].iloc[dev_idx[va]].mean()
    print(f"fold {k}: val_pos_rate={rate:.3f}")

# Fit scaling INSIDE each training fold via a Pipeline -> no target leakage.
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))