Removing Duplicate Lines With Sort, Uniq and Shell Pipes

Use the following syntax:

sort {file-name} | uniq -u

sort file.log | uniq -u

Unix: removing duplicate lines without sorting

The UNIX Bash Scripting blog suggests:

awk ' !x[$0]++'

This command is simply telling awk which lines to print. The variable $0 holds the entire contents of a line and square brackets are array access. So, for each line of the file, the node of the array named x is incremented and the line printed if the content of that node was not (!) previously set.