Lesson 20 of 20 · 8 min
From Clicking to Scripting
Everything up to this point has lived in a browser tab. This module hands you the other half of the skill: pulling the same records by script, so that "look up BRCA1" becomes a line of code you can rerun, share, and trust six months from now.
Why bother automating a click
Clicking through NCBI or EMBL-EBI works fine for one gene. It stops working the moment you need a hundred. Say a collaborator hands you a list of 500 RefSeq accessions and asks for every protein sequence in FASTA — pointing and clicking would take an afternoon and produce no record of exactly what you did. A script does it in seconds and, more importantly, is the record. Anyone who runs it gets the same result. That reproducibility, not raw speed, is the real payoff, and it is why the module ends with a capstone built entirely around it.
The shape of this module
The six lessons climb in altitude, from no code at all to a fully scripted, documented workflow:
| Lesson (this module) | What it adds |
|---|---|
| Searching Web UIs Well (No Code) | field tags and filters that make the browser itself precise |
| Batch Retrieval Without Code | pasting accession lists into NCBI Batch Entrez and EBI's job dispatcher |
| NCBI E-utilities with Biopython | Entrez.esearch/efetch — scripted Entrez, the tool this whole course has been leading up to |
| The Ensembl REST API | JSON over HTTP for genome and transcript data |
| UniProt, KEGG and RCSB PDB APIs | the remaining REST endpoints you will reach for |
| The Formats Databases Emit | reading what comes back — FASTA, GenBank flat file, JSON, XML |
If you have not installed Biopython yet, the install guide walks through it in a few minutes, so the E-utilities lesson is ready to run on day one. The module closes with a capstone, Reproducibility and the BRCA1 Dossier, where you assemble everything scripted along the way — NM_007294, ENSG00000012048, P38398, and the rest of the BRCA1 thread you have followed since What an Accession Number Is — into one rerunnable notebook.
The golden rule: browser first, script second
Never write a query you have not first typed into the web UI. If BRCA1[gene] AND Homo sapiens[orgn] returns the wrong record count in the NCBI website's search box, the identical string will return the wrong count from ESearch too — except now it is buried inside a script and harder to notice. Debug the query where you can see it, then translate it.
This matters because every API in this module is a thin wrapper around the same search logic the website uses. ESearch is Entrez's search box with a URL instead of a form. The Ensembl REST endpoint /lookup/id returns the same fields the Ensembl gene page renders. Getting comfortable in the UI first — which you already are, from The Two Hubs: NCBI Entrez and EMBL-EBI — means every script you write in the next six lessons starts from a query you already know is correct.
One more habit: respect the rate limit
Programmatic access is a shared resource, and every database in this module enforces its own ceiling. NCBI throttles E-utilities to 3 req/s without an API key (10 req/s with one, registered free to your NCBI account), and will start rejecting requests above that. Ensembl allows roughly 55,000 req/hr per client but responds with HTTP 429 and a Retry-After header the moment you cross it — the polite move is to sleep for exactly the seconds that header names, not guess. KEGG caps its REST API at 3 req/s and restricts programmatic use to academic, non-commercial settings; commercial use requires a separate FTP subscription. A loop that fires requests as fast as Python allows will trip one of these limits and get you temporarily blocked — a detail each API-specific lesson covers in full, but worth knowing before you write your first loop.