# Pretrained Language Models for Sequential Sentence Classification

arXiv: https://arxiv.org/abs/1909.04054 · HTML: https://mlsanity.com/paper/1909.04054/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2019.

## Built on by (most cited first, 30 of 53)

- [SciMON: Scientific Inspiration Machines Optimized for Novelty](https://mlsanity.com/paper/2305.14259/) (2023)
- [MS2: Multi-Document Summarization of Medical Studies](https://mlsanity.com/paper/2104.06486/) (2021)
- [Making Science Simple: Corpora for the Lay Summarisation of Scientific Literature](https://mlsanity.com/paper/2210.09932/) (2022)
- [Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation](https://mlsanity.com/paper/2502.05151/) (2025)
- [An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification](https://mlsanity.com/paper/2210.05529/) (2022)
- [Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review](https://mlsanity.com/paper/2204.10805/) (2022)
- [DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions](https://mlsanity.com/paper/2110.08520/) (2021)
- [Efficient Domain Adaptation of Language Models via Adaptive Tokenization](https://mlsanity.com/paper/2109.07460/) (2021)
- [Corpus for Automatic Structuring of Legal Documents](https://mlsanity.com/paper/2201.13125/) (2022)
- [CSFCube -- A Test Collection of Computer Science Research Articles for Faceted Query by Example](https://mlsanity.com/paper/2103.12906/) (2021)
- [CLIP: A Dataset for Extracting Action Items for Physicians from Hospital Discharge Notes](https://mlsanity.com/paper/2106.02524/) (2021)
- [SPEC5G: A Dataset for 5G Cellular Network Protocol Analysis](https://mlsanity.com/paper/2301.09201/) (2023)
- [MASSW: A New Dataset and Benchmark Tasks for AI-Assisted Scientific Workflows](https://mlsanity.com/paper/2406.06357/) (2024)
- [Generating a Structured Summary of Numerous Academic Papers: Dataset and Method](https://mlsanity.com/paper/2302.04580/) (2023)
- [Cross-Domain Multi-Task Learning for Sequential Sentence Classification in Research Papers](https://mlsanity.com/paper/2102.06008/) (2021)
- [Multi-turn Dialogue Reading Comprehension with Pivot Turns and Knowledge](https://mlsanity.com/paper/2102.05474/) (2021)
- [SuperDialseg: A Large-scale Dataset for Supervised Dialogue Segmentation](https://mlsanity.com/paper/2305.08371/) (2023)
- [Enhancing Biomedical Lay Summarisation with External Knowledge Graphs](https://mlsanity.com/paper/2310.15702/) (2023)
- [Comprehensive evaluation of Mal-API-2019 dataset by machine learning in malware detection](https://mlsanity.com/paper/2403.02232/) (2024)
- [Fill In The Gaps: Model Calibration and Generalization with Synthetic Data](https://mlsanity.com/paper/2410.10864/) (2024)
- [Detecting Extraneous Content in Podcasts](https://mlsanity.com/paper/2103.02585/) (2021)
- [KERMIT: Knowledge Graph Completion of Enhanced Relation Modeling with Inverse Transformation](https://mlsanity.com/paper/2309.14770/) (2023)
- [Modeling Multi-level Context for Informational Bias Detection by Contrastive Learning and Sentential Graph Network](https://mlsanity.com/paper/2201.10376/) (2022)
- [A Kind Introduction to Lexical and Grammatical Aspect, with a Survey of Computational Approaches](https://mlsanity.com/paper/2208.09012/) (2022)
- [Understand Legal Documents with Contextualized Large Language Models](https://mlsanity.com/paper/2303.12135/) (2023)
- [Look Ahead Text Understanding and LLM Stitching](https://mlsanity.com/paper/2412.17836/) (2024)
- [Discriminative and Generative Transformer-based Models For Situation Entity Classification](https://mlsanity.com/paper/2109.07434/) (2021)
- [Low-resource Information Extraction with the European Clinical Case Corpus](https://mlsanity.com/paper/2503.20568/) (2025)
- [IdeaReader: A Machine Reading System for Understanding the Idea Flow of Scientific Publications](https://mlsanity.com/paper/2209.13243/) (2022)
- [Enhancing Pre-Trained Language Models with Sentence Position Embeddings for Rhetorical Roles Recognition in Legal Opinions](https://mlsanity.com/paper/2310.05276/) (2023)

## Abstract (arXiv)

As a step toward better document-level understanding, we explore classification of a sequence of sentences into their corresponding categories, a task that requires understanding sentences in context of the document. Recent successful models for this task have used hierarchical models to contextualize sentence representations, and Conditional Random Fields (CRFs) to incorporate dependencies between subsequent labels. In this work, we show that pretrained language models, BERT (Devlin et al., 2018) in particular, can be used for this task to capture contextual dependencies without the need for hierarchical encoding nor a CRF. Specifically, we construct a joint sentence representation that allows BERT Transformer layers to directly utilize contextual information from all words in all sentences. Our approach achieves state-of-the-art results on four datasets, including a new dataset of structured scientific abstracts.