uniq command in Linux
The uniq command can be used to remove the duplicate lines from a text file in Linux or Unix. The duplicate lines should be right next to each other for uniq to work on them, so it is a good idea to first sort the file before passing it through uniq.