From Clicking to Scripting · Subtopic 3 of 7 · 13 min

NCBI E-utilities with Biopython

By the end of this lesson you will be able to pull a named NCBI record — say, the BRCA1 reference transcript NM_007294 — straight into a Python script, with none of the clicking you practiced in the last lesson.

Nine tools, one base URL

The E-utilities are NCBI's programmatic front door to Entrez: nine small web services that all answer at the same base URL, https://eutils.ncbi.nlm.nih.gov/entrez/eutils/. Each utility does one job, and you already met one of them, ELink, in The NCBI Entrez Ecosystem.

Utility	Job
`EInfo`	Lists a database's stats and searchable fields
`ESearch`	Text query → list of UIDs (unique record IDs)
`EPost`	Uploads a UID list to NCBI's server for later reuse
`ESummary`	UID → short document summary
`EFetch`	UID or accession → the full record, in a chosen format
`ELink`	Finds records linked to a UID, in the same or another database
`EGQuery`	Runs one query across all Entrez databases at once, returning counts
`ESpell`	Suggests spelling corrections for a search term
`ECitMatch`	Matches a citation string (journal, year, pages) to a PMID

You can call any of these as a plain HTTP request. In practice, you will almost never do that by hand — you will use Bio.Entrez, the module in Biopython (the Python library for biological data) that wraps all nine calls as ordinary function calls. If you have not installed it yet, the install-biopython guide walks through a clean conda setup.

Telling NCBI who you are

Every Bio.Entrez call needs to carry an identity. Set two module-level variables once, at the top of your script:

python

from Bio import Entrez

Entrez.email = "[email protected]"   # required: how NCBI reaches you if there's a problem
Entrez.api_key = "your-free-api-key"  # optional but strongly recommended

Entrez.email is not optional in spirit — NCBI's usage policy asks for a real address so they can contact you before blocking an IP that is misbehaving, rather than after. The api_key is a free credential you generate from your NCBI account settings, and it changes what you are allowed to do next.

The rate limit, and who enforces it

NCBI caps E-utilities traffic at 3 req/s per source without a key, or 10 req/s with a free API key — tracked per key, with even higher rates available to registered users on request. Recent versions of Bio.Entrez know about this and throttle your calls internally to stay under whichever limit applies, based on whether Entrez.api_key is set.

That automatic throttling covers single calls. It will not save you from a tight loop that fires hundreds of separate EFetch requests for individual IDs — that pattern is slow and impolite even at the legal rate. The better habit, and NCBI's own recommendation, is to batch: pass a comma-separated list of IDs to one EFetch call instead of looping. When you do have to loop, add an explicit time.sleep(0.11) between calls (10 req/s headroom) as a visible guardrail, not just a hope that the library handles it.

Gotcha: an API key raises your rate limit, but it does not turn off NCBI's abuse detection. Hammering the server even at 10 req/s with huge, unbatched loops can still get an IP temporarily blocked. Batch IDs into one call whenever the utility supports it — EFetch and ESummary both accept a comma-separated id list.

Worked example: ESearch then EFetch for BRCA1

The two-step pattern behind almost every script is: ESearch to find the UID(s) you want, then EFetch to pull the full record. Here it is end to end, going after the BRCA1 RefSeq transcript.

python

from Bio import Entrez, SeqIO

Entrez.email = "[email protected]"
Entrez.api_key = "your-free-api-key"

# Step 1: ESearch — find the UID for the BRCA1 RefSeq mRNA
search = Entrez.esearch(
    db="nucleotide",
    term="BRCA1[gene] AND Homo sapiens[orgn] AND RefSeq[filter]",
    retmax=5,
)
result = Entrez.read(search)
search.close()
print(result["IdList"])   # UIDs, not accessions — EFetch can take either

# Step 2: EFetch — pull the GenBank-format record for NM_007294
with Entrez.efetch(db="nucleotide", id="NM_007294",
                    rettype="gb", retmode="text") as handle:
    record = SeqIO.read(handle, "genbank")

print(record.id, len(record.seq), record.description)

# Same accession, FASTA instead of GenBank
with Entrez.efetch(db="nucleotide", id="NM_007294",
                    rettype="fasta", retmode="text") as handle:
    fasta_record = SeqIO.read(handle, "fasta")

print(fasta_record.format("fasta")[:120])

rettype and retmode together choose the output shape: rettype="gb", retmode="text" gives the same annotated GenBank flat file you saw rendered on the website; rettype="fasta" strips it down to header and sequence. EFetch accepts an accession like NM_007294 directly in id — you do not have to resolve it to a numeric UID first, though ESearch is how you would have found that accession in the first place if all you knew was the gene name.

Where this fits, and what is out of scope

NCBI retired its old SOAP-based Entrez interface years ago; every current integration, Biopython included, speaks the REST-style E-utilities shown here. That is a closed question, not a choice to weigh.

This pattern — ESearch to locate, EFetch to retrieve — is the workhorse you will reuse for every NCBI database in this course, from gene to snp to sra. The Batch Retrieval Without Code lesson showed you the no-code version of "fetch many records"; now you can do the same thing under full script control, with retries, rate-limit awareness, and structured objects (SeqRecord) instead of a downloaded file. The BRCA1 dossier you just pulled — one accession, two formats, zero clicks — is the template for scripting any NCBI record you can name.

Batch Retrieval Without Code The Ensembl REST API