How it works
- Step 1Save your Excel sheet as CSV UTF-8 — In Excel, choose
File → Save As → CSV UTF-8 (Comma delimited) (*.csv). The plainCSV (Comma delimited)option writes Windows-1252, which mangles accented names, smart quotes, and CHAR(160).csv-cleanerauto-detects encoding from the BOM, but starting from UTF-8 avoids ambiguity. - Step 2Fork the workflow into your drafts — Click
Forkabove. The route copies the single-nodecsv-cleanergraph (pre-set todedupMode: normalized) into a new private workflow you own, snapshots it as version 1, and records aforkedaudit event. Because the chain is one node, there are no edges to wire — the node is ready to run. - Step 3Drop your CSV onto the node and run — Open the forked draft, drag your CSV onto the
csv-cleanernode's input, and run. Execution happens in your browser — the file is never uploaded to the server. The run panel shows the node go pending → running → done with a row-count summary. - Step 4Decide on case sensitivity —
normalizedlower-cases the comparison key, so it matches Excel's case-insensitivity (Apple=APPLE). If casing is meaningful (e.g.iPhonevsIPHONEare distinct products), changededupModetotrim(whitespace-insensitive, case-preserved) orexact(byte-for-byte). - Step 5Pick the right output line ending —
csv-cleanerdefaults tooutputLineEnding: crlf(what Windows and QuickBooks expect). If you're piping the result into a Unix tool or a databaseCOPY, set it tolf, orpreserveto keep whatever the input used. - Step 6Re-open the output safely — Don't double-click the cleaned CSV — Excel will re-introduce import-side corruption (leading zeros stripped, 16-digit numbers rounded, dates locale-flipped). Open Excel first, then
Data → From Text/CSV(Power Query) and set the long-identifier columns toText, or import into the destination system directly.
The blueprint node chain
The real graph this page forks, straight from lib/orchestrator/seo-workflow-blueprints.ts. It is a single csv-cleaner node — one input, one output, no edges, no schedule. Forking copies exactly this config into a private draft you own.
| Step | Tool (node) | Input → Output | Config applied by the fork |
|---|---|---|---|
| 1 (only) | `csv-cleaner` | csv → csv | trimWhitespace: true · removeEmptyRows: true · repairRows: true · deduplicate: true · normalizeHiddenWhitespace: true · dedupMode: normalized |
Fork / schedule / credential / tier matrix for this workflow
What the orchestrator actually requires to fork, schedule, and run this specific graph. Verified against tier-precheck.ts, the from-blueprint route, audit.ts, and the cron tick route.
| Capability | This workflow | How it works in the engine |
|---|---|---|
| Fork | Any authenticated user | POST /api/orchestrator/workflows/from-blueprint copies the graph into a new private workflow row, snapshots it as v1, and writes a forked audit event |
| Tier to run | **Free** — no gate | csv-cleaner declares no minTier, so precheckWorkflowTier returns required: free, ok: true |
| Credentials | **None needed** | csv-cleaner has no credentialRef field — credentials only apply to connector nodes (Slack, Notion, Google Sheets, etc.), not CSV tools |
| Schedule (cron) | Off by default (scheduleCron: null) | You can add a cron after forking; Cloudflare Cron Triggers hit /api/orchestrator/cron/tick, which enqueues a run for every workflow whose schedule_cron is due since last_fired_at |
| Run history / trace | Yes | Mutations and runs are recorded to the workflow_audit_events table (created, forked, run_triggered, run_cancelled, …) as a fire-and-forget audit trail |
| Where it executes | In your browser | The workflow runner is a client-side module; csv-cleaner processes the file locally and never streams it to the server |
Excel data-corruption bugs that create false duplicates
Excel behaviours that *cause* duplicates that did not exist in the source. If Remove Duplicates produces fewer rows than expected, one of these is usually why. csv-cleaner treats every cell as text, so none of them fire.
| Excel behaviour | What gets corrupted | Why it creates duplicates |
|---|---|---|
| **15-digit precision limit** | Numbers with 16+ digits have digits 16+ replaced with 0 permanently | 4532015112830366 and 4532015112830367 both become 4532015112830360 |
| **Scientific notation conversion** | Numbers with 12+ digits auto-format to 4.53202E+15 | Two 13-digit barcodes display as identical scientific-notation strings |
| **Leading zero stripping** | 00781 becomes 781 | Two distinct SKUs (00781 and 781) collapse to 781 |
| **1,048,576-row truncation** | CSVs larger than the cap are silently truncated on open | Duplicates past the cap are never seen; the result looks correct but is wrong |
| **Auto-correct on dates** | MARCH-1 becomes 1-Mar-2026 | Distinct gene names / product codes turn into dates that match other rows |
Cookbook
Five copy-paste recipes for the situations Excel's Remove Duplicates handles badly: invisible-whitespace duplicates, NBSP from web paste, case-only collisions, leading-zero SKUs, and files past Excel's row cap. Each recipe is the exact csv-cleaner node config the fork drops onto the canvas — adjust one field and run.
Invisible trailing-whitespace duplicates
ExampleThe classic 'Excel says 0 duplicates but I can see them' case. A trailing space on one of two otherwise-identical rows. This is the exact config the fork applies.
csv-cleaner trimWhitespace: true deduplicate: true dedupMode: normalized # input name,email Acme Co,a@acme.test Acme Co ,a@acme.test # <- trailing space, looks identical # output name,email Acme Co,a@acme.test # collapsed: trim runs before the compare
NBSP duplicates from a web / Word paste
ExampleCHAR(160) non-breaking spaces are invisible and Excel treats them as literal characters, so the rows never merge. Turn on hidden-whitespace folding.
csv-cleaner deduplicate: true dedupMode: normalized normalizeHiddenWhitespace: true # folds CHAR(160) + U+200B to space # input (the second 'New York' uses a NBSP between the words) city,count New York,10 New\u00A0York,7 # output city,count New York,10 # NBSP folded to space, then deduped
Case-only collisions (Excel's surprising default)
ExampleExcel merges Apple and APPLE. To replicate that, use normalized. To KEEP them distinct, switch to trim or exact.
# replicate Excel (merge Apple == APPLE) csv-cleaner deduplicate: true dedupMode: normalized # lower-cases the comparison key # keep case-distinct rows (iPhone != IPHONE) csv-cleaner deduplicate: true dedupMode: trim # whitespace-insensitive, case-preserved
Leading-zero SKUs that Excel would corrupt
Example00781 and 781 are different products. Excel strips the zeros and merges them. csv-cleaner keeps cells as text, so the dedupe sees two distinct keys.
csv-cleaner deduplicate: true dedupMode: exact # byte-for-byte; no normalization # input sku,qty 00781,4 781,9 # output: BOTH kept sku,qty 00781,4 781,9 # leading zeros preserved -> not a duplicate
A file past Excel's 1,048,576-row cap
ExampleExcel silently truncates. Run the dedupe in the browser instead. For files that exceed available RAM, split first with csv-row-splitter, dedupe each part, then re-merge.
# direct (fits in RAM) csv-cleaner deduplicate: true dedupMode: normalized # very large file: split -> dedupe -> merge csv-row-splitter (rowsPerFile: 500000) -> csv-cleaner (deduplicate: true, dedupMode: normalized) -> csv-merger (dedupe again to catch cross-chunk dupes)
Edge cases and verbatim errors
Excel reports `0 duplicates found` but you can see them
Whitespace differenceHidden characters: trailing space, leading space, double space, CHAR(160) NBSP, or zero-width U+200B from a copy-paste. Test with =LEN(A1) — different lengths on visually identical cells mean hidden chars. The workflow's trimWhitespace: true + normalizeHiddenWhitespace: true catch all of these before the dedupe compare.
Excel's Remove Duplicates is case-insensitive
Surprising defaultApple and APPLE are merged by Excel. The workflow's default dedupMode: normalized matches that behaviour. If casing matters for your data (product variants, hashes, IDs), switch to trim (case-preserved) or exact (byte-for-byte) so the case-only pairs survive.
1,048,576-row silent truncation
Excel data lossExcel's max worksheet is 1,048,576 rows; larger CSVs are truncated on open with no warning, so duplicates past the cap are never evaluated. The browser runner has no fixed row cap — it is RAM-bound. For files larger than memory, chain csv-row-splitter → csv-cleaner → csv-merger.
15-digit precision corrupts long numbers
Permanent corruptionExcel stores numbers as IEEE-754 doubles, so 16+ digit credit cards / IMEIs / order IDs have their 16th digit onward replaced with 0 — permanent after save, and distinct IDs collapse into false duplicates. csv-cleaner treats every cell as a string, so long identifiers are compared exactly.
Input file has zero data rows (header only or empty)
Empty sourceAn empty CSV or a header-only file has nothing to dedupe. removeEmptyRows: true drops blank lines and the node returns the header (or an empty output) with a zero-rows summary — it does not error. Check the run panel's row count before trusting downstream steps.
Columns differ from what you expected (schema drift)
Schema driftcsv-cleaner dedupes on the whole row, not on named columns — if a re-export added, removed, or reordered columns, two 'same' records may no longer match as full-row duplicates. For column-scoped dedupe (e.g. by email only) use `csv-deduplicator` with explicit match columns instead.
Wrong delimiter (semicolon or tab CSV)
Parse riskEuropean exports often use ; and Amazon/marketplace exports use tab. If the delimiter is mis-read, the whole row lands in one column and dedupe behaves unexpectedly. csv-cleaner auto-detects when delimiter is blank; set it explicitly (;, \t, |) if detection guesses wrong.
Fork attempted while signed out
401 unauthorizedPOST /api/orchestrator/workflows/from-blueprint resolves the caller from the session or an access bearer; with neither it returns 401 unauthorized. Sign in, then fork. Running the forked graph itself stays on the free tier — no upgrade prompt for this single csv-cleaner node.
Output re-opened in Excel re-corrupts the data
User errorDouble-clicking the cleaned CSV re-applies Excel's import coercion, undoing the fix. Open Excel first, then Data → From Text/CSV, set long-identifier columns to Text, or load the file straight into the destination database / app.
Excel dedup error: 'data that has outlines or contains subtotals'
Microsoft-documentedVerbatim Excel error: Excel cannot remove duplicates from data that has outlines or contains subtotals. The workflow never hits this — a CSV carries no outline or subtotal metadata, so csv-cleaner just dedupes the rows.
Frequently asked questions
Why does Excel's Remove Duplicates leave rows that look identical?
Whitespace. Excel's dedup is whitespace-SENSITIVE: Apple and Apple are different rows. But it's case-INSENSITIVE — Apple and APPLE *do* collapse. The combination surprises everyone the first time. Test with =LEN(A1)=LEN(A2) — if FALSE, there's a hidden character. The workflow's trimWhitespace + normalizeHiddenWhitespace catch regular space, CHAR(160) NBSP, and zero-width U+200B before comparing.
Why did Excel's Remove Duplicates delete a row that wasn't actually a duplicate?
Excel corrupted the data on import. Four common causes: (1) 15-digit precision — 16+ digit numbers get rounded so distinct IDs match; (2) leading-zero stripping — 00781 becomes 781; (3) scientific notation — 100000000000123 displays as 1.00000E+14; (4) date locale flip. Use this workflow for any CSV with long numeric identifiers, leading zeros, or mixed-locale dates — csv-cleaner keeps every cell as text.
What exactly does the workflow chain — how many tools?
One. The blueprint is a single `csv-cleaner` node, input csv → output csv, no edges and no schedule. It runs trim, remove-empty-rows, row-repair, and dedupe in a single pass with dedupMode: normalized. There's no multi-tool pipeline to wire — the value is in the configuration Excel can't express.
What does `dedupMode: normalized` actually compare?
It builds the comparison key by trimming, lower-casing, and folding NBSP — but only for the *match*. The cell that survives keeps its original casing and spacing. csv-cleaner also exposes exact, trim (whitespace-insensitive, case-preserved), case-insensitive, and none if you need different semantics.
Does this cost anything — do I need a paid tier?
No. csv-cleaner declares no minTier, so the orchestrator's tier precheck returns required: free, ok: true. Forking and running this graph are both free. (The single node also needs no credentials — those only apply to connector tools like Slack or Google Sheets.)
Is the dedup destructive?
No. The workflow reads your input and writes a new output file; the input is untouched. The run also happens in your browser, so the file is never uploaded. Excel's Remove Duplicates, by contrast, overwrites cells in place and offers no undo after save.
What does the Fork button do?
POST /api/orchestrator/workflows/from-blueprint copies the single-node csv-cleaner graph into a new private workflow you own, snapshots it as version 1, and writes a forked audit event tagged with the blueprint slug. You must be signed in (it returns 401 otherwise). After forking, drop your CSV on the node and run.
Can I schedule this to run automatically?
The blueprint ships with scheduleCron: null (run-on-demand). You can add a cron expression to your forked copy; Cloudflare Cron Triggers call /api/orchestrator/cron/tick, which scans every workflow with a schedule_cron, checks whether it's due since last_fired_at, and enqueues a run. A dedupe is usually triggered by a new export rather than a clock, so on-demand is the common choice.
Is there a run history / audit trail?
Yes. Workflow mutations and runs write to the workflow_audit_events table — created, forked, run_triggered, run_cancelled, run_resumed, and more. It's a fire-and-forget audit log (designed for enterprise compliance), so audit failures never block your run.
My CSV is 1.5 million rows. Why does Excel only show 1,048,576?
Excel's hard maximum; larger CSVs are silently truncated on open. Options: (1) Power Query (Data → Get Data → From Text/CSV) loads to the Data Model with no row limit; (2) this workflow runs in the browser with no fixed cap (RAM-bound); (3) split with `csv-row-splitter`, dedupe each chunk, then re-merge with `csv-merger` and dedupe once more to catch cross-chunk dupes.
I only want to dedupe by one column (e.g. email). Can this do that?
Not directly — csv-cleaner dedupes on the whole row. For column-scoped dedupe, use `csv-deduplicator` with explicit match columns and a keep-first / keep-last strategy. Use this workflow when the goal is whole-row dedupe plus Excel-grade whitespace and type-safety cleanup.
Are there related CSV-cleaning workflows or guides?
Yes. Sibling workflows: Clean an Amazon Seller Central Report (tab-delimited, header-strip, reshape) and Merge & Dedupe Real-Estate Leads (multi-source merge + dedupe). Single-tool guides: remove blank rows from an Excel CSV export, find duplicate invoice numbers, and trim whitespace on an email list.
Local-first by design
This workflow executes entirely on your jadapps-runner. API keys, database credentials, and OAuth tokens are stored in an AES-GCM-encrypted vault on your device — they are never uploaded to JAD Apps' servers. The server only stores the workflow graph (the recipe), not the secrets.