# Transformer over Pre-trained Transformer for Neural Text Segmentation with Enhanced Topic Coherence

arXiv: https://arxiv.org/abs/2110.07160 · HTML: https://mlsanity.com/paper/2110.07160/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 31 references that are in the map.
- First posted on arXiv 14 Oct 2021.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Text Segmentation as a Supervised Learning Task](https://mlsanity.com/paper/1803.09337/) (2018)
- [Text Segmentation by Cross Segment Attention](https://mlsanity.com/paper/2004.14535/) (2020)
- [SECTOR: A Neural Model for Coherent Topic Segmentation and Classification](https://mlsanity.com/paper/1902.04793/) (2019)
- [Two-Level Transformer and Auxiliary Coherence Modeling for Improved Text Segmentation](https://mlsanity.com/paper/2001.00891/) (2020)
- [Improving Context Modeling in Neural Topic Segmentation](https://mlsanity.com/paper/2010.03138/) (2020)
- [Toward Fast and Accurate Neural Discourse Segmentation](https://mlsanity.com/paper/1808.09147/) (2018)

## Built on by (most cited first, 22 of 22)

- [Detecting AI-Generated Sentences in Human-AI Collaborative Hybrid Texts: Challenges, Strategies, and Insights](https://mlsanity.com/paper/2403.03506/) (2024)
- [Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling](https://mlsanity.com/paper/2310.11772/) (2023)
- [Unsupervised Dialogue Topic Segmentation with Topic-aware Utterance Representation](https://mlsanity.com/paper/2305.02747/) (2023)
- [PODTILE: Facilitating Podcast Episode Browsing with Auto-generated Chapters](https://mlsanity.com/paper/2410.16148/) (2024)
- [Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentation](https://mlsanity.com/paper/2112.05364/) (2021)
- [Structured Summarization: Unified Text Segmentation and Segment Labeling as a Generation Task](https://mlsanity.com/paper/2209.13759/) (2022)
- [Recent Trends in Linear Text Segmentation: a Survey](https://mlsanity.com/paper/2411.16613/) (2024)
- [Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.17277/) (2025)
- [Improving Topic Segmentation by Injecting Discourse Dependencies](https://mlsanity.com/paper/2209.08626/) (2022)
- [Topic Segmentation Model Focusing on Local Context](https://mlsanity.com/paper/2301.01935/) (2023)
- [Advancing Topic Segmentation and Outline Generation in Chinese Texts: The Paragraph-level Topic Representation, Corpus, and Benchmark](https://mlsanity.com/paper/2305.14790/) (2023)
- [Interactive Distillation of Large Single-Topic Corpora of Scientific Papers](https://mlsanity.com/paper/2309.10772/) (2023)
- [Team "better_call_claude": Style Change Detection using a Sequential Sentence Pair Classifier](https://mlsanity.com/paper/2508.00675/) (2025)
- [Better Call Claude: Can LLMs Detect Changes of Writing Style?](https://mlsanity.com/paper/2508.00680/) (2025)
- [Topic Segmentation Using Generative Language Models](https://mlsanity.com/paper/2601.03276/) (2025)
- [Unsupervised Scientific Abstract Segmentation with Normalized Mutual Information](https://mlsanity.com/paper/2305.11553/) (2023)
- [Grounded Complex Task Segmentation for Conversational Assistants](https://mlsanity.com/paper/2309.11271/) (2023)
- [Multi-Modal Video Topic Segmentation with Dual-Contrastive Domain Adaptation](https://mlsanity.com/paper/2312.00220/) (2023)
- [A Vietnamese Dataset for Text Segmentation and Multiple Choices Reading Comprehension](https://mlsanity.com/paper/2506.15978/) (2025)
- [Toward General Semantic Chunking: A Discriminative Framework for Ultra-Long Documents](https://mlsanity.com/paper/2602.23370/) (2025)
- [CobSeg: Coherence Boundary Modeling for Dialogue Topic Segmentation](https://mlsanity.com/paper/2605.30668/) (2026)
- [Zero-Shot Cue-Grounded Topic Segmentation of Spoken Documents](https://mlsanity.com/paper/2609.34425/) (2026)

## Abstract (arXiv)

This paper proposes a transformer over transformer framework, called Transformer2, to perform neural text segmentation. It consists of two components: bottom-level sentence encoders using pre-trained transformers, and an upper-level transformer-based segmentation model based on the sentence embeddings. The bottom-level component transfers the pre-trained knowledge learned from large external corpora under both single and pair-wise supervised NLP tasks to model the sentence embeddings for the documents. Given the sentence embeddings, the upper-level transformer is trained to recover the segmentation boundaries as well as the topic labels of each sentence. Equipped with a multi-task loss and the pre-trained knowledge, Transformer2 can better capture the semantic coherence within the same segments. Our experiments show that (1) Transformer2 manages to surpass state-of-the-art text segmentation models in terms of a commonly-used semantic coherence measure; (2) in most cases, both single and pair-wise pre-trained knowledge contribute to the model performance; (3) bottom-level sentence encoders pre-trained on specific languages yield better performance than those pre-trained on specific domains.