Homology, Orthology, and Paralogy

Homology means two sequences descend from a single ancestral sequence. It is a claim about shared history, not a number you read off an alignment. Sequences are either homologous or they are not; there is no such thing as being partly homologous.

Orthologs, paralogs, xenologs

All three are kinds of homolog, distinguished by the event that split them. Orthologs diverged at a speciation event and usually retain the ancestral function, paralogs arose from a gene duplication and are free to subfunctionalize or neofunctionalize, and xenologs owe their relationship to horizontal gene transfer. Repeated duplication and loss are what grow one ancestral gene into a multi-copy gene family, and lineage-specific loss can leave paralogs that masquerade as orthologs, a trap called hidden paralogy.

Why paralogs corrupt trees

A species tree assumes every tip traces the same chain of speciation events. Slip one paralog into the alignment and its history instead records an ancient duplication, so the gene tree can pair the duplicate copy with the wrong species and inflate the branches near the duplication node. Because the offending sequence looks perfectly ordinary, the resulting topology can be strongly supported and still be wrong.

Reciprocal best hits

The reciprocal-best-hit heuristic gives a fast operational proxy for orthology. Gene A in species 1 and gene B in species 2 are called orthologs when A's single best hit in species 2 is B and B's single best hit back in species 1 is A. It is only a heuristic: it misses orthologs whose partner was lost, collapses the many-to-many relationships created by duplication, and can be fooled by fast-evolving copies, so treat its output as candidates rather than truth.

# Build a DIAMOND database from each proteome
diamond makedb --in human.faa -d human
diamond makedb --in mouse.faa -d mouse

# Forward search: every human protein against mouse, keep the top hit
diamond blastp -q human.faa -d mouse -e 1e-5 -k 1 --quiet \
  -f 6 qseqid sseqid bitscore -o human_vs_mouse.tsv

# Reverse search: every mouse protein against human, keep the top hit
diamond blastp -q mouse.faa -d human -e 1e-5 -k 1 --quiet \
  -f 6 qseqid sseqid bitscore -o mouse_vs_human.tsv

# A pair is a reciprocal best hit when it appears in both directions
cut -f1,2 human_vs_mouse.tsv | sort > forward_pairs.txt
awk 'BEGIN{OFS="\t"} {print $2, $1}' mouse_vs_human.tsv | sort > reverse_pairs.txt
comm -12 forward_pairs.txt reverse_pairs.txt

ENSP00000265849	ENSMUSP00000058441
ENSP00000269305	ENSMUSP00000104298
ENSP00000356150	ENSMUSP00000026712
ENSP00000361021	ENSMUSP00000032342

For well-studied taxa you rarely need to compute orthology from scratch. OrthoDB provides a hierarchical catalog of orthologous groups across bacteria, archaea, and eukaryotes, and EggNOG ships precomputed orthologous groups together with functional annotation that eggNOG-mapper can transfer onto your own proteins. Starting from these resources gives you curated groups that already account for the duplications and losses a raw reciprocal-best-hit scan would silently mishandle.