A Gentle Intro to awk and sed

Two small tools handle most everyday text work on the command line: awk and sed. awk is column-aware, so it shines on tables, while sed is a stream editor built for find-and-replace and for deleting lines. Our example is a TSV (tab-separated values) file, where a Tab character separates the columns in each row.

cat genes.tsv

gene    chrom   length  expr
BRCA1   chr17   5589    120
TP53    chr17   2591    340
EGFR    chr7    9905    85
MYC     chr8    1365    410

When awk reads a line, it splits it into fields you name with a dollar sign, so $1 is the first column and $4 is the fourth; a related variable, NF, holds the number of fields on the current line. The first command below uses print to write out two chosen columns, and the comma between them adds a space in the output. To keep only some rows instead, you put a condition in front, so awk '$4 > 100' prints rows whose fourth column is greater than 100. One catch is that the header word in column four is text, not a number, so it can slip through a numeric test, which is why the next command first deletes the header line with sed '1d' and then pipes the result into awk with the | symbol, feeding one command's output straight into the next.

awk '{print $1, $4}' genes.tsv

gene expr
BRCA1 120
TP53 340
EGFR 85
MYC 410

sed '1d' genes.tsv | awk '$4 > 100'

BRCA1   chr17   5589    120
TP53    chr17   2591    340
MYC     chr8    1365    410

sed's most common job is find-and-replace, written as s/old/new/g, where s means substitute and the trailing g means every match on the line rather than just the first. The command below tidies the chromosome labels, turning chr17 into 17 everywhere it appears while leaving the other lines untouched.

sed 's/chr17/17/g' genes.tsv

gene    chrom   length  expr
BRCA1   17      5589    120
TP53    17      2591    340
EGFR    chr7    9905    85
MYC     chr8    1365    410