Shell Data Processing - Uniq ( unique ) - duplicate line removing

Card Puncher Data Processing


uniq stands for “unique” and filters out adjacent, duplicate lines in a file.

It's a filter progam that is part of the Gnu utility package


  • adjacent means that uniq does not detect repeated lines that are not next to each other

You may want to sort the input first, or use 'sort -u' without 'uniq'.

  • comparisons honor the rules specified by 'LC_COLLATE'.



With no options, matching lines are merged to the first occurrence.

Mandatory arguments to long options are mandatory for short options too.
  -c, --count           prefix lines by the number of occurrences
  -d, --repeated        only print duplicate lines, one for each group
  -D                    print all duplicate lines
      --all-repeated[=METHOD]  like -D, but allow separating groups
                                 with an empty line;
  -f, --skip-fields=N   avoid comparing the first N fields
      --group[=METHOD]  show all items, separating groups with an empty line;
  -i, --ignore-case     ignore differences in case when comparing
  -s, --skip-chars=N    avoid comparing the first N characters
  -u, --unique          only print unique lines
  -z, --zero-terminated     line delimiter is NUL, not newline
  -w, --check-chars=N   compare no more than N characters in lines
      --help     display this help and exit
      --version  output version information and exit

A field is a run of blanks (usually spaces and/or TABs), then non-blank characters. Fields are skipped before chars.

Discover More
Bash Liste Des Attaques Ovh
Linux - File

Linux file management See Using Parameters Expansion Removal From a path string where the file does not exist dirname returns the first parent of an existing path file. ...
Card Puncher Data Processing
Shell Data Processing - Filter (Stream Operator)

This page is pipeline operator in a shell language. They are known as filter in a shell language. It is a computer program or shell command (subroutine) that: read from standard input (stream)...

Share this page:
Follow us:
Task Runner