# Text Segmentation by Cross Segment Attention

arXiv: https://arxiv.org/abs/2004.14535 · HTML: https://mlsanity.com/paper/2004.14535/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 30 of 54)

- [Toward Unifying Text Segmentation and Long Document Summarization](https://mlsanity.com/paper/2210.16422/) (2022)
- [Transformer over Pre-trained Transformer for Neural Text Segmentation with Enhanced Topic Coherence](https://mlsanity.com/paper/2110.07160/) (2021)
- [Detecting AI-Generated Sentences in Human-AI Collaborative Hybrid Texts: Challenges, Strategies, and Insights](https://mlsanity.com/paper/2403.03506/) (2024)
- [MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation System](https://mlsanity.com/paper/2503.09600/) (2025)
- [From Text Segmentation to Smart Chaptering: A Novel Benchmark for Structuring Video Transcriptions](https://mlsanity.com/paper/2402.17633/) (2024)
- [Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling](https://mlsanity.com/paper/2310.11772/) (2023)
- [Sequence Model with Self-Adaptive Sliding Window for Efficient Spoken Document Segmentation](https://mlsanity.com/paper/2107.09278/) (2021)
- [MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos](https://mlsanity.com/paper/2306.04216/) (2023)
- [Natural Language Processing for Human Resources: A Survey](https://mlsanity.com/paper/2410.16498/) (2024)
- [Learning Constraints and Descriptive Segmentation for Subevent Detection](https://mlsanity.com/paper/2109.06316/) (2021)
- [Deep Discourse Analysis for Generating Personalized Feedback in Intelligent Tutor Systems](https://mlsanity.com/paper/2103.07785/) (2021)
- [PODTILE: Facilitating Podcast Episode Browsing with Auto-generated Chapters](https://mlsanity.com/paper/2410.16148/) (2024)
- [Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentation](https://mlsanity.com/paper/2112.05364/) (2021)
- [MHMS: Multimodal Hierarchical Multimedia Summarization](https://mlsanity.com/paper/2204.03734/) (2022)
- [Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment](https://mlsanity.com/paper/2210.04722/) (2022)
- [Multi-Granularity Prompts for Topic Shift Detection in Dialogue](https://mlsanity.com/paper/2305.14006/) (2023)
- [Recent Trends in Linear Text Segmentation: a Survey](https://mlsanity.com/paper/2411.16613/) (2024)
- [Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.17277/) (2025)
- [Membox: Weaving Topic Continuity into Long-Range Memory for LLM Agents](https://mlsanity.com/paper/2601.03785/) (2026)
- [Topic Segmentation in the Wild: Towards Segmentation of Semi-structured & Unstructured Chats](https://mlsanity.com/paper/2211.14954/) (2022)
- [Topical Segmentation of Spoken Narratives: A Test Case on Holocaust Survivor Testimonies](https://mlsanity.com/paper/2210.13783/) (2022)
- [SafeRBench: Dissecting the Reasoning Safety of Large Language Models](https://mlsanity.com/paper/2511.15169/) (2025)
- [Towards Multi-Level Transcript Segmentation: LoRA Fine-Tuning for Table-of-Contents Generation](https://mlsanity.com/paper/2601.02128/) (2026)
- [Topic Segmentation Model Focusing on Local Context](https://mlsanity.com/paper/2301.01935/) (2023)
- [Advancing Topic Segmentation and Outline Generation in Chinese Texts: The Paragraph-level Topic Representation, Corpus, and Benchmark](https://mlsanity.com/paper/2305.14790/) (2023)
- [TocBERT: Medical Document Structure Extraction Using Bidirectional Transformers](https://mlsanity.com/paper/2406.19526/) (2024)
- [Beyond Transcripts: A Renewed Perspective on Audio Chaptering](https://mlsanity.com/paper/2602.08979/) (2026)
- [AntPivot: Livestream Highlight Detection via Hierarchical Attention Mechanism](https://mlsanity.com/paper/2206.04888/) (2022)
- [Enhancing Pre-Trained Language Models with Sentence Position Embeddings for Rhetorical Roles Recognition in Legal Opinions](https://mlsanity.com/paper/2310.05276/) (2023)
- [Discovering Elementary Discourse Units in Textual Data Using Canonical Correlation Analysis](https://mlsanity.com/paper/2406.12997/) (2024)

## Abstract (arXiv)

Document and discourse segmentation are two fundamental NLP tasks pertaining to breaking up text into constituents, which are commonly used to help downstream tasks such as information retrieval or text summarization. In this work, we propose three transformer-based architectures and provide comprehensive comparisons with previously proposed approaches on three standard datasets. We establish a new state-of-the-art, reducing in particular the error rates by a large margin in all cases. We further analyze model sizes and find that we can build models with many fewer parameters while keeping good performance, thus facilitating real-world applications.