How to process command output through multiple command-line programs with pipes in Linux

Constructiong Pipelines

A pipeline is a sequence of one or more commands separated by the pipe character (|). A pipe connects the standard output of the first command to the standard input of the next command.

Pipelines allow the output of a process to be manipulated and formatted by other processes before it is output to the terminal. One useful mental image is to imagine that data is “flowing” through the pipeline from one process to another, being altered slightly by each command in the pipeline through which it flows.

Pipeline Examples

This example takes the output of the ls command and uses less to display it on the terminal one screen at a time.

[user@host ~]$ ls -l /usr/bin | less

The output of the ls command is piped to wc -l, which counts the number of lines received from ls and prints that to the terminal.

[user@host ~]$ ls | wc -l

In this pipeline, head will output the first 10 lines of output from ls -t, with the final result redirected to a file.

[user@host ~]$ ls -t | head -n 10 > /tmp/ten-last-changed-files

Pipelines, Redirection, and the tee Command

When redirection is combined with a pipeline, the shell sets up the entire pipeline first, then it redirects input/output. If output redirection is used in the middle of a pipeline, the output will go to the file and not to the next command in the pipeline.

In this example, the output of the ls command goes to the file, and less display nothing on the terminal.

[user@host ~]$ ls > /tmp/saved-output | less

The tee command overcomes this limitation. In a pipeline, tee copies its standard input to its standard output and also redirects its standard output to the files named as arguments to the command. If you imagine data as water flowing through a pipeline, tee can be visualized as a “T” joint in the pipe which directs output in two directions.

Pipeline Examples Using the tee Command

This example redirects the output of the ls command to the file and passes it to less to be displayed on the terminal one screen at a time.

[user@host ~]$ ls -l | tee /tmp/saved-output | less

If tee is used at the end of a pipeline, then the final output of a command can be saved and output to the terminal at the same time.

[user@host ~]$ ls -t | head -n 10 | tee /tmp/ten-last-changed-files

[user@host ~]$ find -name / passwd 2>&1 | less