# Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling

arXiv: https://arxiv.org/abs/2310.11772 · HTML: https://mlsanity.com/paper/2310.11772/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 56 references that are in the map.
- First posted on arXiv 18 Oct 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [CLEAR: Contrastive Learning for Sentence Representation](https://mlsanity.com/paper/2012.15466/) (2020)
- [StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding](https://mlsanity.com/paper/1908.04577/) (2019)
- [Text Segmentation as a Supervised Learning Task](https://mlsanity.com/paper/1803.09337/) (2018)
- [Extractive Summarization of Long Documents by Combining Global and Local Context](https://mlsanity.com/paper/1909.08089/) (2019)
- [Text Segmentation by Cross Segment Attention](https://mlsanity.com/paper/2004.14535/) (2020)
- [SECTOR: A Neural Model for Coherent Topic Segmentation and Classification](https://mlsanity.com/paper/1902.04793/) (2019)
- [Two-Level Transformer and Auxiliary Coherence Modeling for Improved Text Segmentation](https://mlsanity.com/paper/2001.00891/) (2020)
- [Unsupervised Topic Segmentation of Meetings with BERT Embeddings](https://mlsanity.com/paper/2106.12978/) (2021)
- [Improving Unsupervised Dialogue Topic Segmentation with Utterance-Pair Coherence Scoring](https://mlsanity.com/paper/2106.06719/) (2021)
- [Attention-based Neural Text Segmentation](https://mlsanity.com/paper/1808.09935/) (2018)
- [Uncovering the Potential of ChatGPT for Discourse Analysis in Dialogue: An Empirical Study](https://mlsanity.com/paper/2305.08391/) (2023)
- [Transformer over Pre-trained Transformer for Neural Text Segmentation with Enhanced Topic Coherence](https://mlsanity.com/paper/2110.07160/) (2021)
- [Improving Context Modeling in Neural Topic Segmentation](https://mlsanity.com/paper/2010.03138/) (2020)
- [End-to-End Segmentation-based News Summarization](https://mlsanity.com/paper/2110.07850/) (2021)
- [Is Everything in Order? A Simple Way to Order Sentences](https://mlsanity.com/paper/2104.07064/) (2021)
- [Sequence Model with Self-Adaptive Sliding Window for Efficient Spoken Document Segmentation](https://mlsanity.com/paper/2107.09278/) (2021)
- [Structured Summarization: Unified Text Segmentation and Segment Labeling as a Generation Task](https://mlsanity.com/paper/2209.13759/) (2022)

## Built on by (most cited first, 15 of 15)

- [Detecting AI-Generated Sentences in Human-AI Collaborative Hybrid Texts: Challenges, Strategies, and Insights](https://mlsanity.com/paper/2403.03506/) (2024)
- [PODTILE: Facilitating Podcast Episode Browsing with Auto-generated Chapters](https://mlsanity.com/paper/2410.16148/) (2024)
- [Recent Trends in Linear Text Segmentation: a Survey](https://mlsanity.com/paper/2411.16613/) (2024)
- [Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.17277/) (2025)
- [Towards Multi-Level Transcript Segmentation: LoRA Fine-Tuning for Table-of-Contents Generation](https://mlsanity.com/paper/2601.02128/) (2026)
- [CrossFormer: Cross-Segment Semantic Fusion for Document Segmentation](https://mlsanity.com/paper/2503.23671/) (2025)
- [Multimodal Fusion and Coherence Modeling for Video Topic Segmentation](https://mlsanity.com/paper/2408.00365/) (2024)
- [Unsupervised Text Segmentation via Kernel Change-Point Detection on Sentence Embeddings](https://mlsanity.com/paper/2601.18788/) (2026)
- [BP-Seg: A graphical model approach to unsupervised and non-contiguous text segmentation using belief propagation](https://mlsanity.com/paper/2505.16965/) (2025)
- [A Vietnamese Dataset for Text Segmentation and Multiple Choices Reading Comprehension](https://mlsanity.com/paper/2506.15978/) (2025)
- [GOSU: Retrieval-Augmented Generation with Global-Level Optimized Semantic Unit-Centric Framework](https://mlsanity.com/paper/2509.00449/) (2025)
- [Consistent Kernel Change-Point Detection under m-Dependence for Text Segmentation](https://mlsanity.com/paper/2510.03437/) (2025)
- [SegNSP: Revisiting Next Sentence Prediction for Linear Text Segmentation](https://mlsanity.com/paper/2601.03474/) (2026)
- [NLP for Local Governance Meeting Records: A Focus Article on Tasks, Datasets, Metrics and Benchmark](https://mlsanity.com/paper/2602.08162/) (2026)
- [Toward General Semantic Chunking: A Discriminative Framework for Ultra-Long Documents](https://mlsanity.com/paper/2602.23370/) (2025)

## Abstract (arXiv)

Topic segmentation is critical for obtaining structured documents and improving downstream tasks such as information retrieval. Due to its ability of automatically exploring clues of topic shift from abundant labeled data, recent supervised neural models have greatly promoted the development of long document topic segmentation, but leaving the deeper relationship between coherence and topic segmentation underexplored. Therefore, this paper enhances the ability of supervised models to capture coherence from both logical structure and semantic similarity perspectives to further improve the topic segmentation performance, proposing Topic-aware Sentence Structure Prediction (TSSP) and Contrastive Semantic Similarity Learning (CSSL). Specifically, the TSSP task is proposed to force the model to comprehend structural information by learning the original relations between adjacent sentences in a disarrayed document, which is constructed by jointly disrupting the original document at topic and sentence levels. Moreover, we utilize inter- and intra-topic information to construct contrastive samples and design the CSSL objective to ensure that the sentences representations in the same topic have higher similarity, while those in different topics are less similar. Extensive experiments show that the Longformer with our approach significantly outperforms old state-of-the-art (SOTA) methods. Our approach improve F1 of old SOTA by 3.42 (73.74 -> 77.16) and reduces Pk by 1.11 points (15.0 -> 13.89) on WIKI-727K and achieves an average relative reduction of 4.3% on Pk on WikiSection. The average relative Pk drop of 8.38% on two out-of-domain datasets also demonstrates the robustness of our approach.