Remove duplicate lines without losing order

Stable deduplication removes later copies while keeping the first occurrence in place. That distinction matters when a list records priority, discovery order, or the sequence in which options were provided.

The problem

Lists copied from logs, configuration files, imports, or command output often contain repeated lines. A common cleanup goal is to keep the first occurrence while leaving the sequence otherwise untouched.

Order-preserving deduplication

TinyDevTools Remove Duplicate Lines walks the input from top to bottom and keeps the first exact occurrence of each line. This makes the operation stable.

Input:
third
first
third
second
first

Result:
third
first
second

Why not sort first?

A sort changes the sequence before duplicates are removed. That can be useful when alphabetical order is the goal, but it is the wrong transformation when the original sequence represents priority, discovery order, or chronology.

How stable deduplication works

A typical implementation scans from top to bottom and remembers each complete line it has seen. When a line appears again, it skips that occurrence; when it appears for the first time, it emits it immediately. With a hash set, this is an expected linear-time pass, with memory proportional to the distinct lines retained.

For example, a list of preferred mirrors or search results may use the earliest entry as the preferred one. Sorting first would destroy that preference. Stable deduplication keeps each first-seen value in its original position.

Exact matching is a deliberate choice

The TinyDevTools operation compares complete line strings. It does not ignore capitalization, trim spaces, parse fields, or normalize Unicode. This protects meaningful differences, but it also means values that look similar can remain separate:

alpha
alpha
 alpha
Alpha

The first two lines match exactly.
The indented line, capitalized line, and blank line are distinct.

If your task requires case-folding, trimming, or comparing only one field, decide on that normalization first and apply it to a copy. Normalization can merge values that another system considers different.

A safe cleanup workflow

  1. Keep an untouched source copy, especially when the list came from a log or production export.
  2. Find and review repeated lines before removing them if repetition could carry meaning.
  3. Decide whether spaces, case, and blank lines are significant for the receiving system.
  4. Normalize only the distinctions you intentionally want to ignore.
  5. Remove duplicates, then compare the result and line counts with the source.

Lines and records have different duplicate rules

This tool compares the complete text of each line. It does not know where CSV columns begin, which JSON fields identify a record, or whether capitalization in an identifier is significant. A record-oriented task needs two explicit choices: how to parse the format, and which field or fields define equality.

CSV: compare a chosen column after parsing

These sample rows have the same email address under a case-insensitive rule, but different names. The script uses Python's CSV reader so the comma inside "Alex, Jr." remains part of one field. It keeps the first row for each email; using casefold() is appropriate only if the receiving system treats email addresses case-insensitively.

python3 - <<'PY'
import csv
from io import StringIO

source = '''email,name,team
[email protected],"Alex, Jr.",platform
[email protected],Sam,ops
[email protected],"Alex A.",platform
'''
reader = csv.DictReader(StringIO(source))
if "email" not in (reader.fieldnames or []):
    raise ValueError("CSV input must include an email column")
seen = set()
unique_rows = []
for row_number, row in enumerate(reader, start=2):
    email = row.get("email")
    if not email:
        raise ValueError(f"CSV row {row_number} has no email")
    key = email.casefold()
    if key not in seen:
        seen.add(key)
        unique_rows.append(row)

output = StringIO(newline="")
writer = csv.DictWriter(output, fieldnames=reader.fieldnames, lineterminator="\n")
writer.writeheader()
writer.writerows(unique_rows)
print(output.getvalue(), end="")
PY

Output:
email,name,team
[email protected],"Alex, Jr.",platform
[email protected],Sam,ops

The first row wins because the script retains the first occurrence of each normalized key. If case matters, use the original email string instead of calling casefold(). The example rejects a missing header or blank email rather than accidentally grouping such rows together; adapt its validation to the actual data contract.

JSON: compare a selected field, not serialized lines

JSON objects with the same data can have their keys written in a different order. Parse a complete JSON array and compare a field that is actually unique for your use case:

python3 - <<'PY'
import json

source = '''[
  {"id": "u-7", "name": "Alex"},
  {"name": "Alex A.", "id": "u-7"},
  {"id": "u-9", "name": "Sam"}
]'''
records = json.loads(source)
seen_ids = set()
unique_records = []
for record_number, record in enumerate(records, start=1):
    if not isinstance(record, dict) or not isinstance(record.get("id"), str) or not record["id"]:
        raise ValueError(f"JSON record {record_number} must have a non-empty string id")
    record_id = record["id"]
    if record_id not in seen_ids:
        seen_ids.add(record_id)
        unique_records.append(record)

print(json.dumps(unique_records, indent=2))
PY

This keeps the first object for each id, even though the repeated object's keys are in a different order. The example expects a complete JSON array of objects with non-empty string id fields and raises an error for an invalid record; JSON Lines, nested data, and different uniqueness rules need corresponding parsing and validation. Do not normalize identifiers unless the system defining them treats the normalized values as equal.

For chronological logs, repeated events should usually be investigated before removal. If the event identity is a subset of fields, retain timestamps and counts when they matter to the investigation.

Verify the result before replacing data

For either file format, keep the source unchanged, confirm the chosen key against the data contract, and review the retained and discarded records. Check the number of input records, unique keys, and output records; a count mismatch is a reason to stop and inspect malformed or missing keys, not to assume the cleanup succeeded.

Whitespace is data

The current tool compares lines exactly. alpha, alpha with a leading space, and alpha with a trailing space are distinct until you deliberately normalize them.

For shell workflows, awk '!seen[$0]++' file preserves the first occurrence order for ordinary newline-delimited text and compares complete lines. It is not a CSV or JSON parser. If using shell tools, consider locale, file encoding, carriage returns, and whether a final newline is present; keep the original when the transformation matters.

For format-aware processing, see the official Python documentation for the CSV module and JSON module.

Open Remove Duplicate Lines →

To inspect repetition without changing the list, use Find Duplicate Lines. For a sorted set instead, use Sort Lines and then remove duplicates only if the new order is acceptable.

Advertisements