How to Find Duplicate Rows in a CSV File

Sorting doesn't find duplicates, it just puts them next to each other

A common instinct is to sort the file and scan for adjacent identical rows. That works, but it's not what a real duplicate check does: it groups every row by the value of every column and counts how many rows land in each group. Any group with more than one row is a set of exact duplicates — no sorting required, and it also tells you exactly how many extra copies each duplicate has, not just that duplicates exist.

The SQL, in one line

SELECT *, COUNT(*) AS occurrences
FROM data
GROUP BY ALL
HAVING COUNT(*) > 1;

GROUP BY ALL groups by every selected column at once, which is exactly “every column matches” as a query. You can run this yourself in SQL on CSV, or get the same detection (plus severity, count and percentage) automatically from a data quality check.

Where this gets genuinely expensive

Comparing every column of every row against every other row is the one check whose cost scales with the entire dataset at once, not one column at a time — on a 524 MB, 15-million-row file, this query alone measured 14.3 seconds. Past a certain file size, it stops being “a bit slow” and starts risking a genuine out-of-memory crash in the browser tab: a 46.5-million-row Parquet file (523.6 MB — smaller on disk than the CSV above, but far more rows) crashed the query outright during testing, with no error shown and the tool stuck indefinitely. That's a worse failure than slow.

Why a real tool has to say “not calculated” sometimes

Once a file crosses a size and row-count threshold, duplicate detection is skipped outright, and the tool says so — “not calculated, file too large” — rather than silently omitting the row, guessing, or crashing the tab. A byte-size limit alone isn't a reliable guard: Parquet's compression means a file well under a byte-size limit can still hold far more rows than an equivalent CSV, which is exactly the case that caused the crash above. A real row-count limit, alongside the byte-size one, is what actually prevents it.

Once you've found them

A cleaner can remove exact duplicate rows as one explicit, visible step in a pipeline — never automatically from a data-quality finding, always something you added on purpose, with an Original/Cleaned comparison before you export.