Pipes and redirection¶
In 1 minute
- A command prints its results to stdout (normal output) and its errors to stderr.
>saves the output to a file (replacing it).>>adds to the end of the file.|(a pipe) sends one command's output straight into the next command.- Small tools joined with
|answer big questions:grep(keep lines),awk(pick a column),sort,uniq -c(count),head. - In the lab, you answer 20 questions about Bazaar's busiest day, each in one line.
Previously in Bazaar: you learned who may read, change and run each file, and locked down Bazaar's password file.
1. Why learn this?¶
The Diwali sale is over. The boss asks five questions:
- How many requests did Bazaar get today?
- How many checkouts failed?
- Which customer had the most failed checkouts?
- How much money did we make?
- What was the busiest hour?
The answers are all in one file, access.log, with 2,000 lines. You could open it and count by hand for hours.
Or you can answer each question with one line in the terminal, in a second. This lesson shows you how.
2. The concept¶
Bazaar's access log¶
Every request to Bazaar writes one line to logs/access.log:
The parts are separated by spaces. Counting from the left, they are columns 1, 2, 3...:
| Column | Example | Meaning |
|---|---|---|
| 1 | 2026-11-21T09:09:41 |
when |
| 2, 3 | POST /checkout |
what was asked |
| 4 | 500 |
the status code (200 = OK, 500 = failed) |
| 5 | 2941ms |
how long it took |
| 6 | customer=sara |
who |
Two outputs: stdout and stderr¶
Every command has two outputs, both shown on your screen by default:
- stdout ("standard output"): the normal results.
- stderr ("standard error"): the error messages.
They look the same on screen, but they are separate, and you can send them to different places.
Redirection: saving output to a file¶
| You type | What happens |
|---|---|
command > file.txt |
the output goes into file.txt instead of the screen. Replaces the file |
command >> file.txt |
the output is added to the end of file.txt |
command 2> errors.txt |
only the error messages go into errors.txt |
command 2> /dev/null |
the error messages are thrown away (/dev/null is a "black hole") |
Careful with >: like mv in Lesson 2, it replaces the file without asking.
With >> instead, the file keeps both lines.
Pipes: one command's output into the next¶
The | symbol (the pipe, usually above the Enter key) connects two commands:
grep finds the failed checkouts, and instead of printing them, | hands them to wc -l, which counts them.
You see only the final answer.
Think of it as an assembly line in Bazaar's warehouse: one worker picks the right boxes, passes them to the next, who sorts them, who passes them to the next, who counts them. Each worker does one simple job.
The toolbox¶
Five tools do most of the work:
| Tool | Job | Example |
|---|---|---|
grep 'text' |
keep only lines containing the text | grep 'checkout 500' |
awk '{print $6}' |
keep only one column (here, column 6) | the customer |
sort |
put lines in order | equal lines end up together |
uniq -c |
count equal lines that are next to each other | 7 customer=asha |
head -n 3 |
keep only the first 3 lines | the top 3 |
Plus wc -l to count lines, and sort -rn to sort by number, biggest first.
Building a pipeline, one step at a time¶
Question 3: which customer had the most failed checkouts? Build the answer one step at a time,
and look at the output after each step (in the lab, add | head while you're building, so the screen doesn't flood).
Step 1: keep only the failed checkouts.
$ grep 'POST /checkout 500' logs/access.log
2026-11-21T09:09:41 POST /checkout 500 2941ms customer=sara product=masala_box qty=1 amount=0
2026-11-21T09:55:43 POST /checkout 500 2247ms customer=meera product=red_running_shoes qty=3 amount=0
...
Step 2: keep only the customer (column 6).
$ grep 'POST /checkout 500' logs/access.log | awk '{print $6}'
customer=sara
customer=meera
customer=asha
...
Step 3: sort them, so equal names sit together, and count each name.
$ grep 'POST /checkout 500' logs/access.log | awk '{print $6}' | sort | uniq -c
1 customer=arjun
7 customer=asha
2 customer=john
5 customer=meera
2 customer=priya
5 customer=ravi
5 customer=sara
Step 4: biggest first, top 3.
$ grep 'POST /checkout 500' logs/access.log | awk '{print $6}' | sort | uniq -c | sort -rn | head -n 3
7 customer=asha
5 customer=sara
5 customer=ravi
The answer: Asha, with 7 failed checkouts. Bazaar's support team should call her.
That's the whole method: start small, look at the output, add one more | step, look again.
Why sort must come before uniq -c¶
uniq -c only merges lines that are next to each other. Without sort, it counts little runs, not totals:
That's "7 OKs in a row, then 1 failure, then 8 OKs..." Not what we wanted. With sort first, all the 200s are together,
and uniq -c gives real totals.
Questions you may have¶
Where is the | key?
On most keyboards it's above Enter, with \. Press it with Shift.
Why not just write a Python script? For a quick question, one line in the terminal is faster to write than a script. When the logic gets complicated, switch to Python.
Does the pipe make a temporary file? No. The data flows directly from one command to the next, in memory.
My pipeline printed nothing. Why? Usually an early step matched nothing. Remove steps from the end until you see output again; that shows you which step went wrong.
3. See it in action¶
Pick a situation, then press Play.
4. Code example¶
Here are the boss's five questions, each answered with one line, with the real answers from the lab:
# 1. How many requests today?
wc -l < logs/access.log # 2000
# 2. How many checkouts failed?
grep -c 'POST /checkout 500' logs/access.log # 27
# 3. Which customer had the most failed checkouts?
grep 'POST /checkout 500' logs/access.log | awk '{print $6}' | sort | uniq -c | sort -rn | head -n 1
# 7 customer=asha
# 4. How much money did we make? (amount= from successful checkouts)
grep 'POST /checkout 200' logs/access.log | awk -F'amount=' '{s += $2} END {print s}'
# 428291
# 5. What was the busiest hour? (characters 12-13 of each line are the hour)
cut -c 12-13 logs/access.log | sort | uniq -c | sort -rn | head -n 1 # 393 20
So: 2,000 requests, 27 failed checkouts, Asha was hit most often, ₹4,28,291 in sales, and 8 p.m. was the rush hour.
wc -l < file uses <, the opposite of >: it feeds the file into the command. Question 4's awk adds up numbers;
the Going-deeper box below explains it.
5. Under the hood¶
Where the two outputs go¶
$ ls logs /nope > out.txt
ls: cannot access '/nope': No such file or directory
$ cat out.txt
logs:
access.log
The list went into the file. The error still appeared on the screen, because > only redirects stdout.
$ ls logs /nope > out.txt 2> err.txt
$ cat err.txt
ls: cannot access '/nope': No such file or directory
Now each output has its own file, and the screen stays clean.
Going deeper: 2>&1, and why the order matters
2>&1 means "send stream 2 (stderr) to wherever stream 1 (stdout) is going right now".
So the order of the redirections matters:
$ ls logs /nope > all.txt 2>&1 # stdout → file, then stderr → (the file)
$ cat all.txt
ls: cannot access '/nope': No such file or directory
logs:
access.log
$ ls logs /nope 2>&1 > wrong.txt # stderr → (the screen, where stdout is now), then stdout → file
ls: cannot access '/nope': No such file or directory
$ cat wrong.txt
logs:
access.log
In the second line, 2>&1 copied stdout's destination before stdout was moved to the file, so stderr stayed on the screen.
The usual form is > file 2>&1. Bash also has a shortcut for "both into one file": &> file.
Going deeper: awk beyond one column
awk is a small programming language for columns of text.
awk '{print $6}': print column 6.$0is the whole line,NFthe number of columns,$NFthe last one.awk -F'amount=' '{print $2}': split atamount=instead of at spaces, so$2is the number after it.awk '{s += $5} END {print s}': add column 5 of every line intos, and printsonce at the END.awk '$4 == 500 {print $6}': only lines where column 4 is 500.- Arrays:
awk '{n[$6]++} END {for (c in n) print n[c], c}'counts each customer in one step, withoutsort | uniq -c.
Bazaar's revenue per product (task 11 in the lab) uses split() to pull the number out of amount=1298.
Going deeper: more tools — cut, tr, sed, tee, xargs
| Tool | Job | Example |
|---|---|---|
cut -c 12-13 |
keep characters 12 to 13 | the hour from a timestamp |
cut -d= -f2 |
split at =, keep part 2 |
customer=asha → asha |
tr '\n' ' ' |
replace characters (here: newlines with spaces) | join lines into one |
sed 's/customer=//' |
search and replace in every line | remove the customer= prefix |
tee file |
save a copy to a file and pass the data on | grep 500 log \| tee errors.txt \| wc -l |
xargs cmd |
turn input lines into arguments of cmd |
find . -name "*.tmp" \| xargs rm |
Going deeper: how pipes really work
When you run a | b, the shell starts both programs at the same time, as two processes (Module 1, Lesson 2). The kernel connects
a's stdout to b's stdin (standard input, stream 0) through a small buffer in memory (a few tens of KB on Linux). a writes into it,
b reads from it. If b is slower and the buffer fills up, the kernel pauses a until there's room. That's why pipelines can process
files much bigger than your RAM: the data streams through, a little at a time.
And when head -n 3 has its 3 lines, it simply exits. The next write by the command before it fails, and that command stops too. So
sort -rn | head -3 doesn't waste time printing millions of lines nobody will read.
Going deeper: exit codes of a pipeline, and pipefail
A pipeline's exit code is the exit code of its last command. So a failure early in the pipe is hidden:
With pipefail, the pipeline fails if any command in it failed. That's the -o pipefail in every test_lab.sh's
set -euo pipefail (Lesson 1).
6. Lab¶
Lab folder: labs/02-linux-command-line/04-pipes-and-redirection/
cd labs/02-linux-command-line/04-pipes-and-redirection
docker compose up -d
docker compose exec server bash
You land in /srv/bazaar, with logs/access.log: 2,000 requests from Bazaar's Diwali day (the same file for every student).
First you'll try redirection (>, >>, 2>), then the Linux playground: 20 small questions about the log, from "how many lines?"
to "revenue per product". Each task has its answer in the README, so you can check yourself. If you're stuck,
bash /srv/playground/solutions.sh <number> shows one possible solution.
When you finish, run bash test_lab.sh (from your laptop). You should see ALL CHECKS PASSED.
7. Common mistakes¶
>instead of>>. The file is replaced, not added to. Fix:>>to add.uniq -cwithoutsortfirst. You get small runs instead of totals. Fix:sort | uniq -c.- Sorting numbers as text.
printf "9\n10\n" | sort -rputs9before10, because as text "9" > "1". Fix:sort -rn(-n= as numbers,-r= biggest first). - Building a long pipeline in one go. When it's wrong, you can't see where. Fix: add one
|step at a time, and look at the output. - Expecting
>to also catch errors. It only redirects stdout. Fix:2>for errors, or> file 2>&1for both.
8. Quiz¶
9. What's next¶
Next: Environment variables, packages and background jobs. Settings every program can read, installing software, and running commands in the background.
In one line: > and >> save output to files, 2> saves the errors, and | joins small tools (grep, awk, sort, uniq -c, head)
into one-line answers, built one step at a time.
Cheat sheet¶
| Task | Command |
|---|---|
| Save output / add to a file | cmd > f / cmd >> f |
| Save errors / hide errors | cmd 2> f / cmd 2> /dev/null |
| Output and errors into one file | cmd > f 2>&1 |
| Feed a file into a command | cmd < f |
| Count lines | wc -l |
| Lines containing text / not containing / count them | grep 'x' / grep -v 'x' / grep -c 'x' |
| Ignore upper/lower case | grep -i 'x' |
| Column 6 | awk '{print $6}' |
| Count each value | sort \| uniq -c \| sort -rn |
| Top 5 | ... \| head -n 5 |
| Sum a column | awk '{s += $5} END {print s}' |
| Save and pass on | ... \| tee f \| ... |
Next in Bazaar: the database password must reach Bazaar's code without being written in the code. Programs read their settings from environment variables. Let's see how.