uniq command in Linux

The uniq command can be used to remove the duplicate lines from a text file in Linux or Unix. The duplicate lines should be right next to each other for uniq to work on them, so it is a good idea to first sort the file before passing it through uniq.

uniq command examples

Suppose we have a text file “data” as shown below:

# cat data
this is a uniq test
first sort the data before using uniq
uniq can be used to filter out duplicate entries
first to the data before using uniq

Now, passing this file to the uniq command which will NOT remove the duplicate entries:

# uniq data
this is a uniq test
first sort the data before using uniq
uniq can be used to filter out duplicate entries
first to the data before using uniq

See, that the input and output are the same as no adjacent duplicate lines were present in the input file. however, if we first sort the file and then pipe the sorted output to uniq then the results would be different.

# sort data | uniq
first sort the data before using uniq
this is a uniq test
uniq can be used to filter out duplicate entries

Important Uniq Switches

Lets list out the most widely used uniq command switches:

• uniq -u - output only the uniq lines.
• uniq -d - output only the non-uniq i.e. duplicate lines.
• uniq -c - count the number of times each line is repeated.

Let’s see an example of “-c” switch to count the number of times each line is repeated.

# sort data | uniq -c
2 first to the data before using uniq
1 this is a uniq test
1 uniq can be used to filter out duplicate entries

Another very useful switch of uniq is uniq -f which can use to instruct uniq to ignore a certain field while processing the input.