From Clicking to Scripting

From Clicking to Scripting · Subtopic 1 of 7 · 10 min

Searching Web UIs Well (No Code)

This lesson gets you writing precise, repeatable Entrez search strings by hand — the same query syntax that ESearch will send under the hood once you start scripting, so you learn to spot a bad query while you can still see it in a browser tab.

Field tags: telling Entrez which part of the record to match

A plain search box guesses. Type BRCA1 into NCBI's search bar and Entrez checks every indexed field — gene symbols, titles, author names, even incidental mentions — and hands back a noisy pile of results. A field tag fixes that: it is a bracketed suffix that tells Entrez exactly which part of the record your term has to match. BRCA1[Gene Name] only matches records where BRCA1 is the actual gene symbol, not a word that happens to appear somewhere in the text.

Run BRCA1[Gene Name] AND Homo sapiens[Organism] against the gene database and you get exactly one hit: Gene ID 672 — the same record covered in NCBI Gene: the Gene-Centric Hub. Drop the tags and search BRCA1 AND Homo sapiens unqualified, and you pull in every record that merely mentions both terms, which is a much larger and noisier set. The tag is what turns a guess into a lookup.

A few field tags you will use constantly:

Field tagMatchesBRCA1 example
[Gene Name]the gene symbol itselfBRCA1[Gene Name]
[Organism]the source speciesHomo sapiens[Organism]
[Accession]one exact accessionNM_007294[Accession]
[Properties]record metadata like source database or molecule typesrcdb_refseq[Properties]
[Title]words in the record's title linebreast cancer[Title]

Entrez also accepts short aliases for the common ones — [gene] for [Gene Name], [orgn] for [Organism] — which is what you will see in most published queries. Either form works identically.

Boolean operators: AND, OR, NOT

Entrez search strings are boolean expressions, built from AND, OR, and NOT, always in capitals. AND narrows (both conditions must hold), OR widens (either will do), NOT excludes.

text
BRCA1[Gene Name] AND Homo sapiens[Organism]          # narrows to human BRCA1 only
BRCA1[Gene Name] OR BRCA2[Gene Name]                 # widens to either gene
BRCA1[Gene Name] NOT "predicted"[Title]              # excludes computationally predicted records

Operator precedence follows normal boolean rules, and parentheses group terms exactly the way you would expect in any query language: (BRCA1[Gene Name] OR BRCA2[Gene Name]) AND Homo sapiens[Organism] finds either gene, restricted to human. Skip the parentheses on a mixed AND/OR string and Entrez may not group the way you intended — write it out explicitly rather than guessing.

Filters narrow by category, not by identity

A [Properties] filter restricts which kind of record you get back, not which specific record. Search the nucleotide database for BRCA1[Gene Name] AND Homo sapiens[Organism] AND srcdb_refseq[Properties] AND biomol_mrna[Properties] and you get several hundred records — every RefSeq mRNA transcript variant and predicted isoform for the gene, not the single curated reference you might expect. A filter like [Properties] restricts you to a category (here: "RefSeq" and "mRNA"), and a gene this well studied has hundreds of transcript records in that category.

To land on exactly one record, you still need an identity, not a category. NM_007294[Accession] returns precisely one hit — the MANE Select transcript covered in Reading a Prefix: Provenance and Molecule Type. Filters and identity tags do different jobs: filters cut a large set down to a relevant category, an accession tag names one thing.

Try it yourself: search nucleotide for BRCA1[Gene Name] AND Homo sapiens[Organism] AND srcdb_refseq[Properties] AND biomol_mrna[Properties] and note the result count in the corner. Then add AND NM_007294[Accession] to the same string. Watching the count collapse from hundreds to one, live, is the fastest way to feel the difference between filtering and identifying.

The History feature: chaining searches without retyping them

Every search you run on the Entrez website gets a number in the Search History panel — #1, #2, and so on. That is not bookkeeping; it is a reusable handle. Run BRCA1[Gene Name] AND Homo sapiens[Organism] as #1, then run "clinsig pathogenic"[Properties] as #2 in ClinVar, and you can combine them as #1 AND #2 without retyping either string.

Under the hood, each numbered set is backed by a WebEnv (a session token identifying your search history) and a query_key (an index into that history) that NCBI's servers hold on your behalf. You do not see those tokens in the browser — the numbers are the human-readable front end for them — but they are exactly what scripted ESearch/EFetch calls request explicitly with usehistory=y once you start writing code instead of clicking. Getting comfortable chaining #1 AND #2 in the web UI now means the scripted version — passing webenv and query_key between calls instead of re-sending a huge ID list — will feel like the same idea, just typed differently.

Building the BRCA1 query by hand

Put the pieces together and a precise query reads like a sentence: gene, species, category, done. For a clean list of curated human BRCA1 mRNA records, BRCA1[Gene Name] AND Homo sapiens[Organism] AND srcdb_refseq[Properties] AND biomol_mrna[Properties] is that sentence — and because you built it from named, tagged pieces, you can predict what happens when you tighten or loosen any one part of it. That predictability is the whole point: once you translate this exact string into a scripted Entrez.esearch(term=...) call, you will already know what count to expect back.