Adding a method to extract unique rows

[ianengelbrecht]

Such a great tool, all it needs to make it perfect is a 'distinct' or 'unique' command to deduplicate a dataset...

[janxkoci]

+1

In the meantime, you can try one of these:

# awk
xsv cat data.csv | awk '!a[$0]++' > uniq_data.csv
# miller
xsv cat data.csv | mlr -c uniq -a > uniq_data.csv

Awk should be available on most systems, while miller is easy to install, as it's a Go program with zero dependencies. Both are quite fast, but if speed is important, mawk can help a lot.

PS: contrary to the standard Unix command uniq the above tools don't require sorted input and will keep your data in the original order, just deduplicated 😉