Lesson 05 · Getting around

Pipes and text tools: grep, sed, awk

Linux text tools treat text as lines, and every program reads from standard input and writes to standard output. The pipe symbol | joins one program's output to the next one's input, so small tools chain into one command. Follow four lines of scores.txt through grep fail scores.txt | sed 's/fail/retry/' | awk '{ print $1, $3 }': grep keeps the lines that match, sed edits them, and awk splits them into fields and prints the ones you ask for.

10 min10 steps5 questionsBeginner

Fig 5 · Interactive

Loading figure…

or ← →

Check yourself

5 quick questions

Pick an answer to see why. Your score saves with your progress.

01What does the | symbol do in grep fail scores.txt | sed 's/fail/retry/'?

A pipe connects the stdout of the program on its left to the stdin of the program on its right. bash starts both programs together and the kernel passes the text between them.

02scores.txt holds ana 7 pass, ben 3 fail, cy 9 pass and dee 4 fail. What does grep pass scores.txt print?

grep prints only the lines that contain the pattern, here the word pass. (grep -c pass scores.txt would print the count, 2; grep -v pass would print the other two lines.)

03What does sed 's/fail/retry/' do to the line fail fail?

Without the g flag the s command replaces only the first match on each line. s/fail/retry/g would give retry retry.

04In awk, what is $2 for the line dee 4 retry?

awk splits each line into fields at the spaces: $1 is dee, $2 is 4 and $3 is retry. $0 is the whole line.

05After running grep fail scores.txt | sed 's/fail/retry/' | awk '{ print $1, $3 }', what is in scores.txt?

The pipeline only reads the file. sed edits the stream of text flowing through the pipe, not the file; only sed -i writes back into a file, and it overwrites it, so keep a copy first.

Key terms

Glossary

Standard input (stdin): The stream a program reads its input from: stream number 0. Normally the keyboard; in a pipeline, the previous program's output.
Standard output (stdout): The stream a program writes its results to: stream number 1. Normally the terminal; in a pipeline, the next program's input.
Standard error (stderr): Stream number 2, for error messages. It goes to the terminal even in the middle of a pipeline.
Pipe: The | symbol: it joins one program's stdout to the next program's stdin. The kernel keeps a small buffer between them (64 KiB by default on Linux).
grep: A program that prints only the lines that match a pattern.
sed: The stream editor: it changes text as it flows past, for example s/old/new/ to substitute.
awk: A small language for working with fields: it splits each line into $1, $2, ... and runs your program once per line.
Field: One word of a line as awk sees it: by default, fields are separated by spaces or tabs.

Fine print

Notes and sources

grep reads its pattern as a regular expression, a small language for describing text. A plain word such as fail simply matches itself. Useful options: -i ignores upper and lower case, -v keeps the lines that do NOT match, -c only counts them.
Without the g flag (s/fail/retry/g), sed replaces only the first match on each line. Careful: sed -i writes the change back into the file and overwrites it, so try the command without -i first and keep a copy.
On Ubuntu and Debian the awk command is often mawk; Fedora uses GNU awk (gawk). Both run this program the same way. In awk, $0 is the whole line and NF is the number of fields.
A pipe carries stdout only. Error messages go to stderr, which still reaches the terminal, so you see them even in the middle of a pipeline.
The 64 KiB pipe buffer is the Linux default since kernel 2.6.11 (16 pages of 4096 bytes). When it is full, the writer waits; when it is empty, the reader waits.

Quiz Unlocks after the last step.