Remove duplicate lines

When you need a list without duplicates

Duplicates creep in everywhere: email lists merged from several spreadsheets (with the risk of sending the same newsletter twice), SEO keyword lists, discount codes, warehouse SKUs, columns exported from a CSV. This tool always keeps the first occurrence of each line, so the original order of your list never changes, only the later repetitions are dropped. Blank lines never count as duplicates and stay where they are: to remove them there is the «Remove blank lines» box.

Upper and lower case: when to ignore them

For email addresses it's best to turn the option on: JOHN@MAIL.COM and john@mail.com are the same mailbox, so they should count as duplicates. For product codes, passwords, or technical identifiers, on the other hand, case can be meaningful, leave the box unchecked. Either way, the line that is kept preserves its original spelling, completely untouched.

The duplicates a computer cannot see

Two lines can look identical without being so. The commonest case is an invisible space at the start or at the end, and here it does not count: those spaces are ignored in the comparison, together with the zero-width invisible characters that sometimes slip in when copying, and the line that is kept is written exactly as you pasted it. Double spaces in the middle and a tab used instead of a space do count, because sometimes they are part of the data: if you want them not to count, also tick «Remove extra spaces», which runs before the comparison.

The other trap is characters that resemble each other: the straight apostrophe and the curly one, the hyphen and the dash, the Latin «a» and the Cyrillic one. To the eye they are the same line, to a comparison they are two different lines, and no amount of human attention tells them apart. Accented letters written in two ways, the whole «é» or an «e» followed by a separate accent as happens when copying file names from a Mac, are compared as equal instead.

Case and accents: it depends on the data

For email addresses the part after the at sign is case-insensitive, so MARIO@MAIL.COM and mario@mail.com are the same mailbox and should be merged. For a password or a code, by contrast, case matters and merging would destroy the data.

On accents it is nearly always better to distinguish: «pero» and «però» are different words. The exception is lists of personal names typed by different people, where «Nicolo» and «Nicolò» are almost certainly the same person: there the choice is not technical, it is a decision about what the list contains.

Removing duplicates is not always the goal

Sometimes the real question is which lines are repeated and how often, not making them disappear: in a list of orders, a duplicate is an error to understand, not to hide. Deleting it before looking at it loses the most useful information.

It is also worth deciding which copy you keep: the first or the last. If the lines carry a date or a sequence number, the last is usually the good version; if the list is a chronological log, the first. With identical lines it makes no difference, with similar ones it does.

Several clean-ups at once, in a fixed order

Below the box you will find all ten clean-ups of this family, and you can tick as many as you like: each page starts with its own already ticked. The program is the same on all ten, so the same text with the same boxes gives the same result on whichever page you paste it.

The order does not depend on which boxes you tick, and it is the order you see them in: first the characters (emojis, accents, numbers, punctuation), then the spaces inside each line (tabs, indentation, extra spaces), then whole lines (blank and duplicate) and last of all joining everything into one line. So a line that is left empty once the emojis are gone disappears with the blank lines, and two lines that become identical once the spaces are gone count as duplicates.

The sister pages, each with its own box already ticked: Remove extra spaces, Remove indentation, Remove blank lines, Remove emojis, Remove accents, Remove punctuation, Remove numbers, Text on one line and Tabs to spaces.

Nearby tools

To reorder the list before or after there is Sort lines, and to see how long it is Word counter. If the lines come from a CSV, Clean a CSV does the job on columns.