# Cross-Domain Multi-Task Learning for Sequential Sentence Classification in Research Papers

arXiv: https://arxiv.org/abs/2102.06008 · HTML: https://mlsanity.com/paper/2102.06008/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 78 references that are in the map.
- First posted on arXiv 11 Feb 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018)
- [Structural Scaffolds for Citation Intent Classification in Scientific Publications](https://mlsanity.com/paper/1904.01608/) (2019)
- [Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?](https://mlsanity.com/paper/2005.00628/) (2020)
- [PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts](https://mlsanity.com/paper/1710.06071/) (2017)
- [SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications](https://mlsanity.com/paper/1704.02853/) (2017)
- [Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging](https://mlsanity.com/paper/1707.09861/) (2017)
- [Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references](https://mlsanity.com/paper/1402.4578/) (2014)
- [A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks](https://mlsanity.com/paper/1811.06031/) (2018)
- [Pretrained Language Models for Sequential Sentence Classification](https://mlsanity.com/paper/1909.04054/) (2019)
- [Document-Level N-ary Relation Extraction with Multiscale Representation Learning](https://mlsanity.com/paper/1904.02347/) (2019)
- [The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain](https://mlsanity.com/paper/2006.03039/) (2020)
- [Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks](https://mlsanity.com/paper/1703.06345/) (2017)
- [How Transferable are Neural Networks in NLP Applications?](https://mlsanity.com/paper/1603.06111/) (2016)
- [Hierarchical Neural Networks for Sequential Sentence Classification in Medical Scientific Abstracts](https://mlsanity.com/paper/1808.06161/) (2018)
- [Transfer Learning for Named-Entity Recognition with Neural Networks](https://mlsanity.com/paper/1705.06273/) (2017)
- [Multi-Task Learning for Sequence Tagging: An Empirical Study](https://mlsanity.com/paper/1808.04151/) (2018)
- [Domain-independent Extraction of Scientific Concepts from Research Articles](https://mlsanity.com/paper/2001.03067/) (2020)
- [Neural Networks for Joint Sentence Classification in Medical Paper Abstracts](https://mlsanity.com/paper/1612.05251/) (2016)
- [Multi-Task Learning for Argumentation Mining in Low-Resource Settings](https://mlsanity.com/paper/1804.04083/) (2018)
- [A Practitioners' Guide to Transfer Learning for Text Classification using Convolutional Neural Networks](https://mlsanity.com/paper/1801.06480/) (2018)

## Built on by (most cited first, 11 of 11)

- [Harnessing GPT-3.5-turbo for Rhetorical Role Prediction in Legal Cases](https://mlsanity.com/paper/2310.17413/) (2023)
- [Understand Legal Documents with Contextualized Large Language Models](https://mlsanity.com/paper/2303.12135/) (2023)
- [MuLMS-AZ: An Argumentative Zoning Dataset for the Materials Science Domain](https://mlsanity.com/paper/2307.02340/) (2023)
- [Enhancing Pre-Trained Language Models with Sentence Position Embeddings for Rhetorical Roles Recognition in Legal Opinions](https://mlsanity.com/paper/2310.05276/) (2023)
- [Multi-label Sequential Sentence Classification via Large Language Model](https://mlsanity.com/paper/2411.15623/) (2024)
- [SPACE-IDEAS: A Dataset for Salient Information Detection in Space Innovation](https://mlsanity.com/paper/2403.16941/) (2024)
- [Classification and Clustering of Sentence-Level Embeddings of Scientific Articles Generated by Contrastive Learning](https://mlsanity.com/paper/2404.00224/) (2024)
- [Coupling Local Context and Global Semantic Prototypes via a Hierarchical Architecture for Rhetorical Roles Labeling](https://mlsanity.com/paper/2603.03856/) (2026)
- [Pointer-Guided Pre-Training: Infusing Large Language Models with Paragraph-Level Contextual Awareness](https://mlsanity.com/paper/2406.04156/) (2024)
- [Charting the Future of Scholarly Knowledge with AI: A Community Perspective](https://mlsanity.com/paper/2509.02581/) (2025)
- [Semantic Reranking at Inference Time for Hard Examples in Rhetorical Role Labeling](https://mlsanity.com/paper/2605.18007/) (2026)

## Abstract (arXiv)

Sequential sentence classification deals with the categorisation of sentences based on their content and context. Applied to scientific texts, it enables the automatic structuring of research papers and the improvement of academic search engines. However, previous work has not investigated the potential of transfer learning for sentence classification across different scientific domains and the issue of different text structure of full papers and abstracts. In this paper, we derive seven related research questions and present several contributions to address them: First, we suggest a novel uniform deep learning architecture and multi-task learning for cross-domain sequential sentence classification in scientific texts. Second, we tailor two common transfer learning methods, sequential transfer learning and multi-task learning, to deal with the challenges of the given task. Semantic relatedness of tasks is a prerequisite for successful transfer learning of neural models. Consequently, our third contribution is an approach to semi-automatically identify semantically related classes from different annotation schemes and we present an analysis of four annotation schemes. Comprehensive experimental results indicate that models, which are trained on datasets from different scientific domains, benefit from one another when using the proposed multi-task learning architecture. We also report comparisons with several state-of-the-art approaches. Our approach outperforms the state of the art on full paper datasets significantly while being on par for datasets consisting of abstracts.