何気にawkのがすげくね?
awk '!dup[$0]++ {print $0}'