0

Cool AWK trick – remove duplicate lines without sort

I always do something like cat file | sort -n | uniq to get a unique listing of a file to remove duplicates. the only problem with this is that it sorts the list so the ordering of the file is lost.. found a cool trick with AWK today of removing duplicate lines without destroying the order and yet its just a one liner! awk ‘!a[$0]++’ file lets take an example $ cat abc ddd bbb eee fff aaa ccc bbb kkk aaa zzz xxx yyy now : $ awk ‘!a[$0]++’ abc ddd bbb eee fff aaa ccc kkk zzz… Continue Reading