# Semantics of the Unwritten: The Effect of End of Paragraph and Sequence Tokens on Text Generation with GPT2

arXiv: https://arxiv.org/abs/2004.02251 · HTML: https://mlsanity.com/paper/2004.02251/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Apr 2020.

## Built on by (most cited first, 4 of 4)

- [Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation](https://mlsanity.com/paper/2208.11646/) (2022)
- [Improving CTC-based speech recognition via knowledge transferring from pre-trained language models](https://mlsanity.com/paper/2203.03582/) (2022)
- [Finish First, Perfect Later: Test-Time Token-Level Cross-Validation for Diffusion Large Language Models](https://mlsanity.com/paper/2510.05090/) (2025)
- [Song Form-aware Full-Song Text-to-Lyrics Generation with Multi-Level Granularity Syllable Count Control](https://mlsanity.com/paper/2411.13100/) (2024)

## Abstract (arXiv)

The semantics of a text is manifested not only by what is read, but also by what is not read. In this article, we will study how the implicit "not read" information such as end-of-paragraph (eop) and end-of-sequence (eos) affect the quality of text generation. Specifically, we find that the pre-trained language model GPT2 can generate better continuations by learning to generate the eop in the fine-tuning stage. Experimental results on English story generation show that eop can lead to higher BLEU score and lower eos perplexity. We also conduct experiments on a self-collected Chinese essay dataset with Chinese-GPT2, a character level LM without eop or eos during pre-training. Experimental results show that the Chinese GPT2 can generate better essay endings with eop.