Normalise a Multilingual CSV by Stripping Special Characters

A mixed-locale customer table has French, German, and Japanese names plus a stray emoji from a signup form. With Letters on, all the names survive; only the emoji goes.

Input:
id,name
1,José García 🇪🇸
2,Jürgen Müller
3,田中 太郎

Output (all boxes on):
id,name
1,José García 
2,Jürgen Müller
3,田中 太郎

Two records look identical but a zero-width space hides in one, so a multilingual dedup misses it. The stripper removes the invisible character so the values become genuinely equal.

Input (U+200B shown as ·):
id,city
1,Mü·nchen
2,München

These won't match in a dedup.

Output (all boxes on):
id,city
1,München
2,München

Now run /tool/csv-deduplicator to collapse them.

A place name typed with a non-breaking space (common from web copy-paste) loses the NBSP entirely, because only the regular space is kept. The two words run together. This is the case where you should fold NBSP to a space first instead.

Input (NBSP shown as ~):
id,place
1,New~York
2,São~Paulo

Output (all boxes on):
id,place
1,NewYork
2,SãoPaulo

To keep them separate, fold NBSP → space first with
/tool/csv-cleaner, then strip.

French guillemets and curly quotes used around localised phrases are removed, not converted. Decide whether that loss is acceptable or whether you want them folded to straight quotes via csv-cleaner.

Input:
id,phrase
1,“Bonjour”
2,‘Hola’

Output (all boxes on):
id,phrase
1,Bonjour
2,Hola

Translation tools insert soft hyphens (U+00AD) as hyphenation hints. They are invisible but break exact matching. The stripper deletes them, rejoining the word.

Input (soft hyphen shown as ¬):
id,word
1,Zusammen¬arbeit

Output (all boxes on):
id,word
1,Zusammenarbeit

Because the Letters class is \p{L}, accented Latin, CJK, Arabic, Hebrew, Cyrillic, Greek, and other scripts' letters are all kept. This is the core reason to use this tool for multilingual data rather than an ASCII-only stripper.

Only the regular space (U+0020) is kept; the non-breaking space (U+00A0) is removed entirely, so New York written with an NBSP becomes NewYork. If NBSPs are real separators in your data, fold them to regular spaces first with csv-cleaner.

“ ” ‘ ’ « » are removed because they are not in the kept punctuation set. If localised quotation marks carry meaning, normalise them with csv-cleaner instead of stripping.

Older descriptions mention a 'safe mode'; the actual UI has four keep checkboxes (Letters, Digits, Spaces, Punctuation), all on by default. Keeping Letters on is the equivalent of preserving real characters — there is no separate safe-mode switch.

The tool always downloads UTF-8 and does not convert encodings. UTF-8 is the safest choice for multilingual data, but if your target needs UTF-16 or a legacy codepage, convert after download with a dedicated encoder.

Precomposed accented letters (NFC, e.g. a single é) are single letters and are kept. Decomposed forms (NFD: base letter + combining accent) keep the base letter and the combining mark, which is also a \p{L}-adjacent mark category — verify rare decomposed data renders correctly after stripping.

Localised or English column names in the first row are never modified, including any invisibles in them. If a header carries a BOM or zero-width character, clean it separately with csv-find-replace.

International data still includes IDs, prices, and dates. Turning Digits or Punctuation off removes those characters globally (19,99 → 1999 or worse). Keep both on for normalisation; use csv-find-replace for targeted numeric edits.

Free is 2 MB / 500 rows; Pro is 100 MB / 100,000 rows. Large multilingual catalogues exceed free — split with csv-row-splitter or upgrade.

Arabic and Hebrew letters are kept, but explicit bidirectional control marks (LRM/RLM, U+200E/U+200F) are not letters and are removed. This usually has no visual effect but can change directionality in rare mixed-direction strings — verify RTL columns after stripping.