Fix Chinese CSV encoding without damaging the data

When Chinese names turn into symbols or strings such as 中文, the text usually has not “changed language.” The bytes were decoded with the wrong character encoding. The safe fix is to identify the source encoding, decode once, and write a separate verified output.

What mojibake actually means

A CSV file is a sequence of bytes. An encoding tells software how those bytes map to characters. UTF-8 represents Unicode text with a variable number of bytes. GB18030 is a Chinese national character encoding widely encountered in exports from Windows and older business systems. Both can represent Chinese text, but their byte sequences are interpreted differently.

Mojibake appears when the reader guesses a different encoding from the writer. Renaming the file, changing .csv to .txt, or choosing another font does not convert the bytes. Repeatedly opening and saving the damaged display can make recovery harder because the wrong characters may be written back as new data.

Start with evidence, not trial-and-error saving

Keep an untouched copy and inspect the first few kilobytes in a capable text editor. If the file begins with the UTF-8 byte-order mark, that is a useful signal. A file without a mark may still be UTF-8. If strict UTF-8 decoding fails and the file came from a mainland Chinese Windows application, GB18030 is a reasonable candidate to test.

An encoding detector can suggest candidates, but it cannot understand whether a decoded customer name is correct. Validate with known Chinese phrases, uncommon characters, punctuation, and a few records whose expected values are available. A successful decode with no software error is necessary, but readable and semantically correct samples are stronger evidence.

Convert by decoding once and encoding once

The conversion has two explicit steps: read bytes using the confirmed source encoding, then write text using the chosen destination encoding. For a Python workflow, strict error handling prevents silent substitution:

from pathlib import Path

source = Path("export-original.csv")
target = Path("export-utf8-bom.csv")

text = source.read_text(encoding="gb18030", errors="strict")
target.write_text(text, encoding="utf-8-sig", errors="strict")

Use utf-8 instead of gb18030 when that is the verified source. The destination utf-8-sig writes UTF-8 with a BOM, which is often more reliably recognized by Microsoft Excel on Windows. This example converts text only; it does not validate CSV columns, quoted line breaks, duplicates, or spreadsheet formula risks.

Preserve identifiers as text

Encoding repair and data-type conversion are separate operations. A spreadsheet may display a correctly decoded value while still changing 001234 into 1234, converting a long identifier to scientific notation, or guessing a date. Customer IDs, postal codes, telephone numbers, invoice references, and product codes should remain text unless a documented rule says otherwise.

Opening the CSV through an import dialog can let you select UTF-8 and mark identifier columns as text. If another system consumes the file, test the new CSV in that exact importer. A file that looks correct in a text editor can still be interpreted differently by a spreadsheet or database tool.

Validate CSV structure after conversion

Character conversion should not change the logical table, but validation still matters. Use a real CSV parser rather than splitting lines at commas. Quoted fields may contain commas, quotation marks, or line breaks. Check that:

• the header and expected column count remain unchanged;
• the record count matches the source;
• Chinese names and addresses are readable;
• quoted commas stay inside their original fields;
• leading-zero identifiers remain intact;
• no replacement character such as � appears unexpectedly;
• the original file hash and modification time remain untouched.

Sample both common and unusual records. Include the first and last rows, a record with uncommon Chinese characters, a quoted address, and an identifier with leading zeros. Keep a small report recording the detected source encoding, chosen destination encoding, input and output counts, and any additional cleanup rule.

When UTF-8 BOM is useful

A BOM is not required for valid UTF-8, and many programming tools work without it. In Windows spreadsheet workflows it can act as an explicit hint and reduce incorrect legacy-encoding guesses. The correct choice depends on the receiving application. If a downstream system forbids a BOM, export plain UTF-8 and document that requirement instead.

Do not treat a BOM as proof that every row is valid. It identifies the intended Unicode encoding, but it says nothing about delimiters, quoting, record counts, or business rules. Reopen the exported file with a strict CSV parser and the target spreadsheet application before replacing any operational import.

Use a local tool when the file is private

Customer exports may contain personal or commercial information. Uploading them to an unknown converter creates another copy and another party handling the data. CSV Desk runs locally in Microsoft Edge, supports UTF-8 and GB18030 input, writes a separate UTF-8 BOM CSV, preserves text identifiers, and produces a JSON change report. For files requiring a custom duplicate rule or a reviewed service workflow, the fixed-scope CSV cleanup service starts with a public inquiry containing only fictional examples.

Disclosure: this guide and the linked software were produced with AI assistance and browser-tested. Encoding selection still requires verification against the source system and representative records.