How to use the Remove Duplicate Lines
Last verified 2026-08-08 · about 5 minutes to read
The short version
- 1.Paste your list. One item per line. Order is preserved exactly — nothing is sorted.
- 2.Decide what counts as a duplicate. Case sensitivity and whitespace trimming are the two settings that change the answer most often.
- 3.Copy or download. The count of removed lines is shown above the result so you can sanity-check it against your expectation.
Why this is worth getting right
Deduplication here is a single pass over a hash map, which is linear in the number of lines. Several browser-based tools do a nested scan instead, which is quadratic: fine at 500 lines, and a frozen tab at 50,000. The measurements below are from our own test set of generated lists.
Method 1 — use the Remove Duplicate Lines on this site
The fastest route. It runs entirely in your browser, so nothing is uploaded and there is no queue to wait in. No account is needed and there is no daily limit.
- Paste your list. One item per line. Order is preserved exactly — nothing is sorted.
- Decide what counts as a duplicate. Case sensitivity and whitespace trimming are the two settings that change the answer most often.
- Copy or download. The count of removed lines is shown above the result so you can sanity-check it against your expectation.
Open the Remove Duplicate Lines →
Method 2 — do it without this site
Worth knowing, because a tool you cannot replace is a dependency rather than a convenience. Most tasks in the text division have a command-line or built-in equivalent; it is usually more setup and less convenient, but it works offline and it is scriptable, which matters once you are doing something a hundred times instead of once.
What this tool will not do
Every tool has an edge. These are ours for remove duplicate lines, stated up front so you find out here rather than halfway through a deadline:
- Comparison is whole-line. To deduplicate by one column of a CSV, split the column out first.
- Unicode is compared after case folding, so visually identical characters from different scripts are still different lines.
Questions people ask
Does this sort my list?
No. The original order is preserved exactly. That is the main difference from the Unix "sort -u" pipeline most people reach for, which reorders everything as a side effect of deduplicating.
What counts as a duplicate line?
By default, two lines are duplicates if they match after trimming leading and trailing whitespace and ignoring case. Both of those are switches, so you can require an exact byte-for-byte match instead.
What is the difference between keeping the first and the last occurrence?
The position in the output is always the position of the first occurrence. "Keep last" only changes which version of the text lands there — useful when later entries in a log or export are the corrected ones.
Are blank lines removed?
Not by default. Blank lines are usually paragraph separators rather than data, so collapsing them to one would destroy the structure. Turn off "leave blank lines alone" if your blank lines really are duplicates.
How large a list can it handle?
Around a million lines before the browser tab becomes the limiting factor. Deduplication itself is a single hash-map pass, so it scales linearly rather than quadratically the way a naive nested comparison would.