How to Translate a Large TXT File: Encoding, Chapters, and Line Breaks

To translate a large TXT file, prepare three things before you upload it anywhere. First, make sure the text is decoded correctly and save it as UTF-8. Second, put every chapter heading on its own line with a blank line around it. Third, if the file breaks lines at a fixed width, join those lines so that each paragraph is one line, with a blank line between paragraphs. Then translate the whole file, and afterwards compare paragraph and chapter counts between the source and the translation.

A plain text file has no styles, no headings, and no metadata. Everything a translation tool knows about its structure comes from the bytes and the line breaks. That is why the same three problems keep coming back: the bytes are read in the wrong encoding, the chapters cannot be told apart from ordinary lines, or the line breaks say nothing about where a sentence ends.

We made sample files in four encodings, wrote a free checker, and translated a hard-wrapped short story into Simplified Chinese nine times with a local model. The model handled the hard wrapping fine when it saw the whole file. It broke sentences and gave one character's action to another in all three runs where the file was cut into pieces by line count.

Quick Checklist

The checker reports on every row and fixes the encoding, byte order mark, line endings, and hard wrapping. Chapter headings and missing blank lines are yours to fix by hand. Run python3 check_txt.py inspect yourfile.txt to see the report, and python3 check_txt.py convert yourfile.txt clean.txt --unwrap to write a UTF-8 copy with the paragraphs joined. It needs only Python 3, and it never changes the original file.

Step 1: Get the Encoding Right

A TXT file is a sequence of bytes. The encoding is the rule that turns those bytes into characters, and the file does not say which rule was used. If a program guesses wrong, you get mojibake: text that is garbled, or worse, text that looks fine but is not what was written.

Here is the same Chinese phrase, 最后一班渡轮 ("the last ferry"), read with the wrong rule. We generated these with Python's built-in codecs, and you can reproduce them with the sample files.

The last row is the dangerous one. Japanese bytes read as Chinese produce real Chinese characters, with no error and no �. A translation tool will happily translate them, and the result will be nonsense with good grammar.

Our checker shows why you cannot rely on "it opened without errors." When we ran it on the Shift_JIS sample, two encodings decoded the whole file without a single error:

encoding: unknown (not UTF-8)
these legacy encodings decode without errors; read the previews,
because a wrong one can still produce real-looking characters:
  gb18030  戞堦榖丂嵟屻偺僼僃儕乕
  cp932    第一話　最後のフェリー

Only the second line is Japanese. The first is a string of rare Chinese characters that no one wrote. On the GB18030 Chinese sample, five encodings decoded without errors, and only one of them, GB18030, showed the chapter heading 第一章 最后一班渡轮. So the rule is simple: decide the encoding by reading the first lines, not by whether a program complained.

Where the guesses usually go wrong:

• Chinese text from older Windows systems or Chinese sites is often GBK or GB18030. GB18030 covers everything GBK does, and the WHATWG Encoding Standard, which browsers follow, goes further: it says "GBK's decoder is gb18030's decoder." If a file is GBK, opening it as GB18030 works.
• Traditional Chinese from Taiwan or Hong Kong is often Big5.
• Japanese is often Shift_JIS, or Microsoft's extended version of it, which Windows and Python call cp932.
• Korean is often EUC-KR.
• A file full of NUL bytes, or one where every other character looks empty, is usually UTF-16 without a byte order mark.

A byte order mark (BOM) is a short invisible code at the start of a file that announces UTF-8, UTF-16, or UTF-32. When one is present, trust it. The same standard says that "the byte order mark is more authoritative than anything else." Our checker reads the BOM first and only guesses when there is none.

Save the file as UTF-8

Once the text reads correctly, save it as UTF-8. The WHATWG standard calls UTF-8 "the most appropriate encoding for interchange of Unicode," and it is the safest choice for any translation tool.

There is a second reason that matters for translation in particular. Legacy encodings only cover the characters of their own region. Shift_JIS has no code for 这, the first character of 这是最后一班渡轮, and Big5 has no code for it either, because it is a Simplified Chinese character. Windows-1252 cannot store Cyrillic. If the translated file has to be written back in the original legacy encoding, the target language may not fit. BookTranslator's TXT translator says it auto-detects common encodings and returns the translated file "with the same encoding as the input," so starting from UTF-8 removes the question entirely.

You can convert in a text editor (open with the right encoding, then "Save as" UTF-8), with iconv -f GB18030 -t UTF-8 in.txt > out.txt on macOS or Linux, or with the checker:

python3 check_txt.py convert book.txt book-utf8.txt --encoding gb18030

If the file already contains � characters, the damage happened before you got it. Converting again will not restore the missing characters. Look for a better copy of the source.

Step 2: Make Chapters Easy to Find

Chapters matter for three reasons. A long file is translated in pieces, and chapter breaks are the cleanest places to cut. You will want to check each chapter after translation. And if you later turn the text into an ebook, the chapter headings become the table of contents.

Put each heading on its own line, with a blank line before and after it, and use one pattern throughout the book. These are the patterns our checker counts:

• Chapter 1, Chapter One: The Last Ferry, Part Two, Prologue, Epilogue
• 第一章, 第12回, 第三話, 第二卷
• a number or a Roman numeral alone on a line, such as 12 or XII.

Run inspect and read the list of headings it found. If the book has 40 chapters and the checker finds 37, three headings are hiding: one runs into the first sentence, one uses a different word such as "Ch. 7," or one has text on the same line. Fix those three by hand. It takes less time than finding the problem in a translated file.

Step 3: Unwrap Hard Line Breaks, Carefully

Some TXT files break every line at a fixed width, so that the text fits a plain terminal or a narrow window. Text copied out of PDFs and OCR output does the same thing at the width of the printed page. In these files a line break means "the line was full," not "the sentence or paragraph ended."

Hard wrapping is easy to spot. In our hard-wrapped sample, the checker reported:

non-empty lines: 43; longest line: 66 chars; 25 lines within 12 chars of the longest
line breaks inside paragraphs that do not end a sentence: 23
looks hard-wrapped: consider convert --unwrap

The fix is to join the lines inside each paragraph and keep the blank lines between paragraphs. The trap is text whose line breaks are real: verse, song lyrics, letters, addresses, recipes, and lists. Our sample includes a five-line letter:

My dear Mara,
The key is under the blue pot.
Do not let your uncle sell the boat.
With love, always,
Nonna

A blunt "replace every single line break with a space" turns that into one run-on line. The checker's --unwrap only joins a paragraph when most of its lines run close to the full width, so short-line blocks like this letter are left alone. On our sample it rebuilt the original unwrapped file exactly, byte for byte, letter included. On your own file, check a few joined paragraphs and every block of verse before you trust it.

Two more details. When you join lines of Chinese or Japanese text, do not add a space between them; the checker joins CJK lines, including lines that end in CJK punctuation, without one. And if your text came from OCR, words may also be split with hyphens at the ends of lines, which needs its own decisions. Our guide to fixing OCR line breaks and hyphenation covers that.

What We Tested: Hard-Wrapped Text, Whole vs. Cut Up

We wanted to know whether hard wrapping actually hurts the translation, or only looks untidy. We wrote a 396-word short story for the test: three chapters, fifteen paragraphs, dialogue, the letter above, and a * * * scene break. We saved it twice with the same words: unwrapped, with one paragraph per line, and hard-wrapped at 66 characters, which has 43 non-empty lines instead of 19.

Then we asked a local model to translate it into Simplified Chinese three ways, three runs each:

1. Wrapped, whole file. The hard-wrapped file in one request.
2. Unwrapped, whole file. The unwrapped file in one request.
3. Wrapped, cut every 12 lines. The hard-wrapped file split every 12 lines, one request per piece, the replies joined back together. This is what happens when a script or tool splits a long file by line count instead of by paragraph.

The model was Qwen 3.6 27B (Q4_K_M) in LM Studio, with thinking turned off and temperature 0.3, on an Apple M4 Max with 64 GB of memory. Every request used the same one-sentence prompt. We ran it on October 28, 2026. The script is run_llm_txt_test.py, all nine outputs are published unedited, and results.csv has one row per run.

The first finding surprised us. When the model saw the whole hard-wrapped file, it joined the lines itself. All three outputs had one paragraph per line, the same 19 lines as the unwrapped runs, and read just as well. So hard wrapping alone did not hurt this model on this short file.

Cutting the file by line count did hurt, in the same places every time. Three of the 12-line pieces ended in the middle of a sentence. The first ends at "He pointed with his" and the next piece starts with "pen at a three-storey building on the corner." Translated separately, the second half lost its subject. In two runs it came back as 她瞥了一眼街角那栋三层楼的建筑, "she glanced at the three-storey building on the corner." In the third, it ended with 随后她重新回到填字游戏上, "then she went back to the crossword." In the story, the man in the ticket booth points and goes back to his crossword. In all three runs, a reader of the Chinese would think Mara did it.

The second cut, between "the careful, slightly old-fashioned hand she had learned at a" and "convent school," left the Chinese sentence hanging at 那是她在, "that was where she," in all three runs (one run added a stray 一), and the word "learned" disappeared. The third cut split the sentence about the shutters across two lines, and the second half turned "painted, not well, but not long ago" into peeling paint: 油漆虽已斑驳 in two runs and 油漆剥落 in the third. And because the pieces were joined with a single line break, the blank line before the scene break vanished, which is why each run had 14 paragraphs instead of 15. Our checker caught the paragraph count. It could not catch the wrong character; we found that by reading.

What this means for your file:

• Unwrapping protects you from splitters you do not control. Every long file is split somewhere, whether by your own script, a chat window's limit, or the tool you upload to. If paragraphs are single lines separated by blank lines, any reasonable splitter can cut between paragraphs instead of inside a sentence.
• The worst errors were fluent. "She glanced at the building" is good Chinese. A reader would notice the odd line breaks at each cut and the dangling 那是她在, but nothing marks the wrong character or the peeling paint as mistakes. Errors at split points are meaning errors first, formatting errors second.
• This was one model and one short story. A file this short fits in one request, so the whole-file runs never had to split anything. A full novel does have to be split, which is exactly when the preparation in Steps 1 to 3 pays off. These files test preparation and splitting, not BookTranslator's own pipeline.

One smaller observation: across the six whole-file runs, the model wrote Mara's name as 玛拉 in five and 马拉 in one. Each output was consistent with itself, but separate requests did not agree with each other. For a long book translated in many pieces, a list of names and their agreed spellings keeps every chapter in line. Our glossary guide shows how to build one.

Step 4: Translate, Then Check the Counts

When the file is clean, translating it is the easy part. You can upload it to BookTranslator's TXT translator, which accepts .txt, .md, and .mdx files, says it handles TXT files up to 300 MB, and shows the price before checkout. There is no free full translation, but the upload itself tells you the cost. If you prefer to work with a chat model or a local LLM, split the file at chapter or paragraph boundaries yourself and keep a glossary for names. If your text is Markdown rather than plain prose, see our Markdown translation guide, because Markdown syntax needs its own rules.

After translation, compare the two files:

python3 check_txt.py compare book-utf8.txt book-translated.txt

If either file is not UTF-8, add --source-encoding or --translated-encoding. The chapter check knows English and Chinese/Japanese heading patterns; for other target languages, add the local word for "chapter" to the pattern at the top of the script.

This is the report for one of our cut-up runs:

                            source  translated
paragraphs                      15          14  <- differs
chapter headings                 3           3
non-empty lines                 43          24  <- differs
scene breaks (* * *)             1           1
replacement chars                0           0
source script LATIN characters left in translation: 0

Read it like this:

• Paragraphs and chapter headings should match. A missing paragraph often means two were merged or one was dropped.
• Non-empty lines can differ for a good reason. Our whole-file wrapped runs went from 43 lines to 19 because the model joined the hard-wrapped lines, which was an improvement. Compare lines only when the source was already unwrapped.
• Source-script characters left over point to passages that were not translated at all. A few are normal for names, product terms, or quotations; a run of them is a skipped section.
• Replacement characters in the translation mean an encoding problem on the way out.

Counts cannot tell you whether the translation is right. Read the first and last paragraph of each chapter, and the paragraphs on either side of every place the file was split. That is where our test found every serious error.

Frequently Asked Questions

Why does my TXT file look fine in one program and garbled in another?

The two programs guessed different encodings. One guessed right. Find the encoding that shows the text correctly, then save the file as UTF-8 so every program reads it the same way.

Should I convert a TXT file to EPUB or DOCX before translating?

Only if you want what those formats add. A clean TXT file with clear chapters translates well as it is. If you want a table of contents, chapter navigation, and an ebook you can read on a device, our guide to turning a web novel into a translation-ready EPUB walks through it.

Can I translate a TXT file with mixed languages?

Yes, but tell the translator what to leave alone. Quotations or terms in a third language, and names that should stay in their original script, are easier to protect with a glossary or a short instruction than to fix afterwards.

What about a TXT file I downloaded from a website?

Check that you have the right to use it. For books in the public domain or files you wrote yourself, the steps above apply. A file's encoding or format tells you nothing about its copyright.