Remove duplicate lines

Deduplication takes a list and returns it with repeated entries removed. It sounds trivial, and the only difficult part is deciding what counts as “the same” — which is entirely a judgement about your data, not about the algorithm.

What counts as a duplicate

The default is an exact match: two lines are the same if every character matches. That is strict, and it means all of these survive as distinct entries:

• hello and Hello — different case
• hello and hello — trailing space
• hello and hello — leading space
• hello world and hello world — double space

For machine-generated data, exact matching is usually right. For anything typed by a human, it usually is not — email lists, survey responses and address data are full of entries that differ only in case or stray whitespace.

Two options change the answer:

Trim whitespace before comparing. Almost always what you want. Leading and trailing spaces are invisible and virtually never meaningful.

Ignore case. Right for email addresses (the domain part is definitively case-insensitive, and in practice the local part is treated that way by every major provider), usernames and tags. Wrong for passwords, code identifiers, and any data where case carries meaning.

Order and which copy survives

Standard deduplication keeps the first occurrence of each entry and removes later ones, preserving the original order. That matters when the order encodes something — priority, time, or a manual sort.

If your data has been sorted first, “first occurrence” becomes arbitrary among identical entries. That is fine when the lines are truly identical and matters a great deal when they differ in ways your comparison is ignoring — deduplicating case-insensitively keeps whichever casing happened to come first, which may not be the one you wanted.

Practical uses

• Cleaning email lists before an import, where duplicates cost money per record and can trigger sending penalties.
• Tidying exported data from systems that append rather than update.
• Deduplicating keyword lists gathered from several sources.
• Cleaning log files to see the distinct set of errors rather than their volume.
• Merging two lists — paste both, deduplicate, and you have the union.

Where people go wrong

Deduplicating structured data by line. A CSV where one row is genuinely a duplicate of another except for an ID column will not be caught, because the lines differ. Line-based deduplication only works when the whole line is the unit of identity. For CSV, deduplicate on the key column in a spreadsheet or database instead.

Losing count information. If you needed to know that an entry appeared 14 times, deduplicating destroys that. Count first.

Ignoring case on data where case matters. Deduplicating a list of code identifiers case-insensitively will silently merge userId and UserID, which may be two different things.

Assuming visually identical means identical. Non-breaking spaces, different Unicode dash characters, and smart versus straight quotes all look the same and compare as different. If two lines appear identical but survive deduplication, one of those is usually why — the text diff tool will show you which characters differ.

Not keeping the original. Deduplication is not reversible. Keep a copy until you have confirmed the result is what you expected.

Your text is processed entirely in your browser and never transmitted — see the privacy policy.