0

Cool AWK trick – remove duplicate lines without sort

I always do something like cat file | sort -n | uniq to get a unique listing of a file to remove duplicates. the only problem with this is that it sorts the list so the ordering of the file is lost..

found a cool trick with AWK today of removing duplicate lines without destroying the order and yet its just a one liner!

awk '!a[$0]++' file

lets take an example
$ cat abc
ddd
bbb
eee
fff
aaa
ccc
bbb
kkk
aaa
zzz
xxx
yyy

now :
$ awk '!a[$0]++' abc
ddd
bbb
eee
fff
aaa
ccc
kkk
zzz
xxx
yyy

enjoy

jlim0930

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.