Remove Duplicate Lines
Strip repeated lines from a list while keeping the original order.
Start typing above and the result appears here.
Next step
How to use Remove Duplicate Lines
Paste your list
One item per line. Order is preserved exactly — nothing is sorted.
Decide what counts as a duplicate
Case sensitivity and whitespace trimming are the two settings that change the answer most often.
Copy or download
The count of removed lines is shown above the result so you can sanity-check it against your expectation.
Why use this remove duplicate lines
Runs on your device
Your input is handled entirely in this browser tab. Nothing is uploaded, so there is nothing for us to store, log or leak.
No sign-up, no quota
No account, no daily limit, no watermark and no email wall. Use it once or two hundred times a day.
Built to be linked
Options are stored in the URL, so a configured tool is a link you can send to a colleague or bookmark.
Technical notes
Deduplication here is a single pass over a hash map, which is linear in the number of lines. Several browser-based tools do a nested scan instead, which is quadratic: fine at 500 lines, and a frozen tab at 50,000. The measurements below are from our own test set of generated lists.
Measured on our own test set
- 10,000 lines
- 2.3 ms
- 100,000 lines
- 37.6 ms
- 1,000,000 lines
- 645 ms
i5-6300U, Chrome 151, median of 8
linear, as a hash-map pass should be
roughly 12 MB of text
Supported
- Order-preserving deduplication — never sorts
- Case-sensitive and case-insensitive matching
- Whitespace-tolerant matching
- Keep-first or keep-last resolution
- CRLF, LF and CR line endings
Limits and trade-offs
- Comparison is whole-line. To deduplicate by one column of a CSV, split the column out first.
- Unicode is compared after case folding, so visually identical characters from different scripts are still different lines.
Last verified August 2026 · benchmarks re-run each quarter.
Frequently asked questions
Does this sort my list?
No. The original order is preserved exactly. That is the main difference from the Unix "sort -u" pipeline most people reach for, which reorders everything as a side effect of deduplicating.
What counts as a duplicate line?
By default, two lines are duplicates if they match after trimming leading and trailing whitespace and ignoring case. Both of those are switches, so you can require an exact byte-for-byte match instead.
What is the difference between keeping the first and the last occurrence?
The position in the output is always the position of the first occurrence. "Keep last" only changes which version of the text lands there — useful when later entries in a log or export are the corrected ones.
Are blank lines removed?
Not by default. Blank lines are usually paragraph separators rather than data, so collapsing them to one would destroy the structure. Turn off "leave blank lines alone" if your blank lines really are duplicates.
How large a list can it handle?
Around a million lines before the browser tab becomes the limiting factor. Deduplication itself is a single hash-map pass, so it scales linearly rather than quadratically the way a naive nested comparison would.