Clean a Web-Scraped HTML Table CSV

An HTML table cell <td>Population&nbsp;1,234,567</td> becomes the string 'Population\xa01,234,567' in the resulting CSV. The character between Population and 1 is U+00A0, not U+0020. Downstream string comparisons (== 'Population 1,234,567') fail. The cleaner's normalizeHiddenWhitespace folds it. Pandas' own documented recipe (unicodedata.normalize('NFKC', x).strip()) does the same thing.

Source HTML:
  <td>Country</td><td>Population&nbsp;(2024)</td>

After pandas.read_html() + to_csv() (showing \xa0 as ·):
Country,Population·(2024)
France,67,500,000·(est.)

Cleaner config: normalizeHiddenWhitespace: true

Output:
Country,Population (2024)
France,67,500,000 (est.)

Wikipedia auto-formats quotes in article-body content. Tables embedded in articles inherit the styled characters. "Lord of the Rings" becomes “Lord of the Rings”. Downstream NLP / search-indexing pipelines treat the two forms as different strings.

Source (Wikipedia table cell, after read_html):
Title,Year
“The Lord of the Rings”,1954
“Foundation”,1951

Cleaner config: normalizeSmartQuotes: true

Output:
Title,Year
"The Lord of the Rings",1954
"Foundation",1951

An older non-English source page declared as <meta charset="latin-1"> (or with no charset declaration, sniffing as Latin-1) passes through pandas/BeautifulSoup as Latin-1 strings. Saving with to_csv(encoding='utf-8') then re-reading produces mojibake on accented characters. The cleaner detects the actual encoding from the BOM (or content heuristics) and re-decodes correctly.

Source HTML (declared Latin-1 charset):
<td>José</td>

After scrape → save as utf-8 (mojibake):
Name
José

Cleaner with inputEncoding: auto re-decodes correctly:
Name
José

Wikipedia tables include inline footnote references. The cleaner doesn't strip these — that's outside its scope (text content vs whitespace/encoding). For Wikipedia-specific cleanup, chain with csv-find-replace.

Source (typical Wikipedia data table):
Population,Source
"67,500,000[1]",United Nations[2]
"60,200,000[citation needed]",

Cleaner output (whitespace + quotes normalised, footnotes preserved):
Population,Source
"67,500,000[1]",United Nations[2]
"60,200,000[citation needed]",

Follow-up: csv-find-replace with regex (?i)\[\d+\]|\[citation needed\]
  → empty for footnote removal.

pandas.to_csv('out.csv') writes UTF-8 without a BOM. Excel-on-Windows falls back to Windows-1252 → mojibake. The cleaner adds the BOM, equivalent to pandas' encoding='utf-8-sig'.

Source (after pandas.to_csv defaults):
Name
José Müller

Excel-on-Windows display (Windows-1252 fallback):
Name
José Müller

Cleaner adds UTF-8 BOM on output → Excel auto-detects → displays correctly.

A table with merged cells (colspan/rowspan) that pandas read_html misparsed produces rows of unequal length. The cleaner's repairRows: true pads short rows to the header width — but the structural relationships are lost. For correct extraction, parse with a colspan/rowspan-aware library (the tabulate-html Python package, or a custom BeautifulSoup loop).

Source table with colspan=2 in row 2 misparsed by read_html:
Country,Region,Population
France,Europe,67500000
Germany,83200000
  ↑ row 3 only has 2 cells because pandas didn't expand the rowspan

Cleaner with repairRows: true pads to 3 columns:
Country,Region,Population
France,Europe,67500000
Germany,,83200000
  ↑ now 3 columns but the Region value is empty — needs source-side fix

Quoted verbatim from the pandas reference: 'read_html attempts to push the idiosyncrasies of the HTML contained in tables to the user'. The cleaner is the cleanup pandas docs expect you to do. The canonical recipe — unicodedata.normalize('NFKC', x).strip() applied to every cell — does exactly what normalizeHiddenWhitespace: true + trimWhitespace: true (default) does, just in one drop-on-file pass instead of an applymap() call.

  is HTML 4 / HTML 5 syntax for U+00A0 (non-breaking space). Every modern HTML parser decodes it to the actual U+00A0 codepoint — which is then preserved as a real character through CSV, JSON, every database column. Visually identical to U+0020 regular space; never matches in exact-string comparison. The cleaner's normalizeHiddenWhitespace is built specifically for this.

Some older / non-English source pages declare Latin-1 (<meta charset="latin-1">) or no charset at all (legal but ambiguous). pandas defaults to Latin-1 detection in some paths; the resulting strings are Latin-1 codepoints. Saving as encoding='utf-8' (without re-encoding) produces a file where each Latin-1 byte was reinterpreted as the same Unicode codepoint — mojibake. Fix at source: pass encoding='utf-8' and flavor='lxml' or flavor='html5lib' to read_html for better detection. Fix at output: the cleaner's inputEncoding: auto re-decodes if the file genuinely was saved as Latin-1; mojibake from a misdeclared save needs scraper-side fix.

Per scraper-community consensus, pandas read_html handles ~85% of merged-cell tables correctly but breaks on complex multi-level headers, tables split across multiple <tbody> sections, or rowspan values that extend beyond the visible table. Result: CSV rows of unequal length. The cleaner's repairRows: true pads short rows to header width but the underlying relationships are lost (cells appear in the wrong columns). For complex tables, use a colspan/rowspan-aware library (tabulate-html, custom BeautifulSoup) at scrape time, then run the cleaner for whitespace/encoding.

Wikipedia and similar wiki sources embed [edit] links, [citation needed] superscripts, and footnote references ([1], [2]) as inline text. These survive the scrape as cell content. The cleaner doesn't strip them — that's a text-replacement task, not a whitespace/encoding task. Chain with csv-find-replace using regex \[\d+\]|\[edit\]|\[citation needed\] → empty, with caseSensitive: false.

An HTML table with two <thead> rows (group header + subheader) produces a pandas MultiIndex column header. Saving to CSV writes the headers as Python tuple strings: "('Country', 'Name')", "('Population', '2024')". The cleaner doesn't auto-flatten these — use csv-header-rename with a renames JSON to consolidate into single-level headers, OR re-export with pd.read_html(url)[0].columns = pd.read_html(url)[0].columns.map('_'.join) before to_csv.

.get_text(strip=True) strips leading and trailing whitespace from the concatenated text. It does NOT collapse internal whitespace and does NOT fold U+00A0 to U+0020. The string 'Population\xa02024' keeps the U+00A0. The cleaner's normalizeHiddenWhitespace handles it post-scrape.

Open in pandas' issue tracker since 2019: <br> tags inside table cells are not converted to whitespace or a newline — pandas concatenates the surrounding text together, often losing the visual separation. A cell with Apples<br>Oranges<br>Pears may parse as ApplesOrangesPears. The cleaner can't recover the lost separation — fix at scrape time by pre-processing the HTML (e.g. html.replace('<br>', '\n') before passing to read_html) or by using a custom BeautifulSoup loop that calls .get_text(separator='\n') which DOES respect <br>.

Some browser 'Save table as CSV' extensions copy the cell's innerHTML rather than innerText — producing <a href="...">Link</a> as literal cell content. The cleaner doesn't strip HTML — that's outside its scope. Either use a different extension that exports innerText, or chain with csv-find-replace using regex <[^>]+> → empty for HTML tag stripping.

Scraping a paginated table (page 1, page 2, ...) and concatenating the outputs without explicit dedup produces duplicate rows when source pages overlap. The cleaner's dedupMode: exact (default) collapses byte-identical rows. For near-duplicates (case differences, whitespace), use dedupMode: trim or normalized.