How to Remove Duplicate Lines Without Losing Order

You paste a keyword list into a spreadsheet, hit "remove duplicates", and two rows that are obviously the same both survive. One of them has a space at the end. You cannot see it, the software can, and you can lose ten minutes hunting for something invisible.

For a simple one-item-per-line list, a dedicated text tool can be more straightforward than cleaning spreadsheet cells, as long as you know what it counts as "the same". This guide walks through the Remove Duplicate Lines tool using one small list we actually ran through it, including the parts that surprised us.

What the tool does

You paste a list, one item per line. The tool keeps the first copy of every line and drops the later ones. The order of what survives is the order you pasted, so a list you sorted by hand or by priority stays that way. There is nothing to submit: the result box updates as you type, and three small counters under it show lines in, lines out and lines removed.

There are three checkboxes: Ignore case, Trim spaces and Remove empty lines. When we opened the page, only Trim spaces was ticked. That matters, because the same list gives different answers depending on the boxes.

The test list and what came out

This is the exact seven-line list we typed in. Line 4 has a single trailing space, and line 5 is blank.

red running shoes
Red Running Shoes
running shoes women
red running shoes␣
(blank line)
running shoes women
trail running shoes

Here is what each combination of boxes produced. Nothing was edited by hand afterwards.

Boxes ticked	Lines left	What stayed
None	6 of 7	Only the second "running shoes women" went. The trailing-space line and the capitalised line both stayed, and so did the blank line.
Trim spaces	5 of 7	The line with the trailing space was now recognised as a repeat of line 1 and removed.
Trim spaces + Ignore case	4 of 7	"Red Running Shoes" went as well. The three keywords remained, along with the blank line.
All three	3 of 7	Clean: red running shoes, running shoes women, trail running shoes.
Remove empty lines only	5 of 7	The blank line went, but the trailing-space line and the capitalised line were still there.

The row worth remembering is the first one. With no boxes ticked, the tool treats a line as a repeat only when every character matches, spaces included. "red running shoes" and "red running shoes " are different strings, and the tool says so.

Small things we checked

• Which spelling survives with Ignore case? The first one you pasted. In an email list where Ben@Example.com came before ben@example.com, the capitalised line stayed and the lowercase one went. The tool does not rewrite anything to lowercase, so the spelling you keep depends on the order you pasted.
• Does Trim spaces change the lines you keep? Yes, it trims them. A line pasted with two spaces on each side came out as padded line, with no leading or trailing spaces.
• Tabs inside a line. Trim only cleans the two ends. In our test, item1[tab]item1 and item1[tab][tab]item1 were kept as two different lines, because the second has an extra tab in the middle.
• Non-breaking spaces. A line ending in a non-breaking space (the kind that sneaks in from web pages and Word) was matched with the plain version when Trim spaces was on, and kept separate when it was off.
• Windows line endings. A list with CRLF endings was cleaned like any other, and the result came out with ordinary line breaks.

How to use it

1. Open the tool and paste your list into the Input box, one item per line. You can also use Open .txt file.
2. Decide on the three boxes. For keyword lists, we would tick Trim spaces and Ignore case. For email lists, Ignore case is usually practical, but check whether capital letters matter in your data. For case-sensitive things like file names on Linux, or code, leave it off.
3. Look at the counters. If "Removed" is lower than you expected, check for differences in capital letters, spaces at the ends of lines, or characters in the middle of the line.
4. Press Copy, or Download .txt. The download is a plain text file called unique-lines.txt and contains only the result.

One honest note on the plan side: downloads of images and PDFs made as a guest or on the free plan carry a small Toolysite watermark, while paid plans do not. This tool outputs plain text, and the file we downloaded had no watermark in it.

Where it slows down

We also tried big lists. A list of 20,000 lines with about 8,000 different values came back in a couple of seconds on our test PC. A list of 100,000 lines took close to a minute, and the page felt stuck while it worked. That was one machine and one browser, so your numbers will differ, but these tests suggest it is fine for a few thousand lines at a time and a poor choice for a full database export. For very large lists, use a command-line tool, or work in smaller batches and then run the combined result through the tool once more, because duplicates that sit in different batches will survive the first pass.

Tips from the test

• If lines still look identical after cleaning, turn on Trim spaces first and Ignore case second. Those two explain most leftovers.
• Want the list alphabetical afterwards? Remove duplicates first, then run the result through the Text Sorter. Doing it the other way round works too, but then you lose your original order.
• For messy spacing in the middle of lines, the Whitespace Cleaner collapses repeated spaces, which this tool does not.
• Keep your original list somewhere. The tool keeps the first copy only. There is no option to keep the last.

Frequently asked questions

Why were two identical-looking lines not removed?

Almost always a trailing space, a different capital letter, a non-breaking space or a tab in the middle. Tick Trim spaces and Ignore case first. If a pair survives both, check for a tab or double space inside the line.

If I ignore case, which capitalisation is kept?

Whichever version appears first in your list. The tool does not convert anything to lowercase. If you want one consistent style, run the cleaned list through the Case Converter afterwards.

Does Trim Spaces remove extra tabs inside a line?

No. It only trims the start and end of each line. Spaces or tabs in the middle of a line count as part of the text, so two lines that differ only there are treated as different.

Can I safely remove duplicates from 100,000 lines at once?

It works, but slowly. We measured roughly a minute for 100,000 lines against a couple of seconds for 20,000, on one computer. Smaller batches are kinder to the page.

If you write a lot, you might also like our guide to title case and sentence case, which covers another text clean-up job where the automatic result needs a quick human check.