Wildcards: Acting on Many Files at Once

A single sequencing run can leave you with dozens of files whose names differ by only a number or a letter, such as gzip-compressed FASTQ read files ending in .fastq.gz. A wildcard is a special character that stands in for other characters, so one short pattern can act on many files at once. Before your command runs, the shell replaces the pattern with the real matching file names, a step called globbing.

ls -F        # list this folder; the -F flag puts a slash after folder names

backup/  control.fastq.gz  notes.txt  sample_1.fastq.gz  sample_10.fastq.gz  sample_2.fastq.gz  sample_3.fastq.gz

The star wildcard

ls *.fastq.gz    # * matches any run of characters, so this lists every file ending in .fastq.gz

control.fastq.gz  sample_1.fastq.gz  sample_10.fastq.gz  sample_2.fastq.gz  sample_3.fastq.gz

Matching single characters

ls sample_?.fastq.gz       # ? matches exactly one character, so sample_10 (two characters) is skipped
ls sample_[123].fastq.gz   # [123] matches one character that is 1, 2, or 3
# both commands print: sample_1.fastq.gz  sample_2.fastq.gz  sample_3.fastq.gz

Curly braces { } do something different, called brace expansion. The shell writes out every option in the list, turning one word into several, so sample_{1,2}.fastq.gz becomes two separate names. This is pure text expansion: unlike the star, braces do not check whether any matching files exist, so they still work for names that do not exist yet. That makes them ideal with mkdir to create several new folders at once, and a compact way to name several existing files, for example when copying them with cp.

cp sample_{1,2}.fastq.gz backup/    # the shell expands this to: cp sample_1.fastq.gz sample_2.fastq.gz backup/
mkdir -p results/{trimmed,aligned}  # makes results/trimmed and results/aligned; -p also creates the parent results folder