How Unix sort and uniq work together
The common sort | uniq pipeline works because uniq compares neighboring lines. Understanding that step helps you choose between a sorted set, an occurrence count, and first-seen deduplication.
What uniq actually does
Unix uniq primarily collapses adjacent identical lines. If identical lines are separated, a plain uniq does not remove them.
alpha beta alpha uniq keeps all three. sort | uniq alpha alpha beta becomes alpha beta
Why sorting changes the problem
sort groups equal lines so uniq can collapse them. The tradeoff is that original order is discarded. That is often useful for set-like data but not for chronological or priority-ordered data.
Choose an explicit output
For a line-oriented file where sorted order is acceptable, these GNU/Linux commands illustrate different results. They assume GNU coreutils and an existing names.txt; the C locale makes the ASCII example's ordering reproducible:
printf '%s\n' beta alpha beta > names.txt LC_ALL=C sort -- names.txt | uniq LC_ALL=C sort -- names.txt | uniq -c LC_ALL=C sort -u -- names.txt
The first command outputs alpha and beta; the count form reports one alpha and two beta lines; the last outputs the same sorted unique values. GNU uniq counts or removes adjacent identical lines; sorting groups equal lines first. Count-column spacing is for alignment and can vary with the largest count.
sort has options that affect comparison, locale, and numeric behavior. The examples use lowercase ASCII; for mixed-case or non-ASCII data, explicitly choose the ordering your workflow requires and consult your local manual. These options are shown for GNU/Linux and may need adjustment on other Unix-like systems. See the GNU uniq manual for exact option behavior.
When plain uniq is enough
If duplicates are already adjacent and you want to collapse only consecutive repeats, plain uniq preserves the order of the remaining groups. It does not find equal values separated by other lines. This can be useful for run-length style summaries, where transitions and group boundaries matter.
Input: red red blue red uniq output: red blue red
Protect source data and choose semantics
Do not redirect output over the same source file you are reading. Write to a separate file, inspect the output, and replace the source only when you are certain. For a stable first-seen result, use a stable-deduplication method instead of sorting; for investigation, count or find duplicates before changing anything.
Choosing the workflow
- Use stable deduplication when first-seen order matters.
- Use sorting when canonical order is the desired output.
- Use a duplicate finder when you need to inspect repetition before changing anything.
Related: preserve first-seen order in the browser or learn the surrounding Linux text-file commands.
Advertisements