Lesson 6 of 14 · 10 min

Pipes and Redirection: Connecting Commands

Every command-line program communicates with the world through three built-in channels called standard streams. Once you understand these streams you can steer a program's output into a file, feed a file into a program, or connect several programs into one flowing pipeline. This lesson walks you through each of those moves and ends with a small pipeline of your own.

A program reads its input from standard input, usually called stdin, which by default is whatever you type on the keyboard. It writes its normal results to standard output, called stdout, and it writes error messages to a separate channel called standard error, or stderr. Both stdout and stderr appear on your screen by default, but they are kept separate so you can handle real results and error messages differently.

bash

echo "BRCA1" > genes.txt
echo "TP53" >> genes.txt
echo "EGFR" >> genes.txt
echo "TP53" >> genes.txt
cat genes.txt

BRCA1
TP53
EGFR
TP53

The echo command prints the text you give it, and the single greater-than sign, >, sends that output into genes.txt, creating the file and replacing anything already in it. The double greater-than sign, >>, instead appends a new line to the end of the file, and the cat command prints a file's contents to the screen. Error messages travel on the separate stderr channel, and the symbol 2> catches them, where the 2 is simply the number the shell uses for standard error.

bash

cat missing_file.txt 2> errors.txt
cat errors.txt

cat: missing_file.txt: No such file or directory

Building your first pipeline

A pipe, written as the vertical bar |, connects two commands by sending the standard output of the command on its left straight into the standard input of the command on its right. This lets you chain small tools so data flows through them one after another. In the command below the less-than sign < feeds genes.txt into sort, the pipe passes the sorted lines to uniq, and > saves the final result to a file.

bash

sort < genes.txt | uniq > unique_genes.txt
cat unique_genes.txt

BRCA1
EGFR
TP53

Try it yourself: Build a file called samples.txt containing the four lines control, treated, control, treated, using echo with > for the first line and >> for the rest. Then run sort < samples.txt | uniq -c > counts.txt followed by cat counts.txt to see how often each sample name appears. The -c option tells uniq to count the repeats of each line, and sorting first is what lets uniq group the identical lines together.

Permissions and Making Scripts Executable Wildcards: Acting on Many Files at Once