# MARRO: Multi-headed Attention for Rhetorical Role Labeling in Legal Documents

arXiv: https://arxiv.org/abs/2503.10659 · HTML: https://mlsanity.com/paper/2503.10659/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 72 references that are in the map.
- First posted on arXiv 8 Mar 2025.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Active Example Selection for In-Context Learning](https://mlsanity.com/paper/2211.04486/) (2022)
- [FinBERT: A Pretrained Language Model for Financial Communications](https://mlsanity.com/paper/2006.08097/) (2020)
- [Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents](https://mlsanity.com/paper/2105.03887/) (2021)
- [LEGAL-BERT: The Muppets straight out of Law School](https://mlsanity.com/paper/2010.02559/) (2020)
- [ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation](https://mlsanity.com/paper/2105.13562/) (2021)
- [Identification of Rhetorical Roles of Sentences in Indian Legal Judgments](https://mlsanity.com/paper/1911.05405/) (2019)
- [Exploring the Use of Text Classification in the Legal Domain](https://mlsanity.com/paper/1710.09306/) (2017)
- [Semantic Segmentation of Legal Documents via Rhetorical Roles](https://mlsanity.com/paper/2112.01836/) (2021)
- [Corpus for Automatic Structuring of Legal Documents](https://mlsanity.com/paper/2201.13125/) (2022)
- [Train Once, Test Anywhere: Zero-Shot Learning for Text Classification](https://mlsanity.com/paper/1712.05972/) (2017)
- [Few-Shot Fairness: Unveiling LLM's Potential for Fairness-Aware Classification](https://mlsanity.com/paper/2402.18502/) (2024)
- [Legal Case Document Similarity: You Need Both Network and Text](https://mlsanity.com/paper/2209.12474/) (2022)
- [Legal Judgment Reimagined: PredEx and the Rise of Intelligent AI Interpretation in Indian Courts](https://mlsanity.com/paper/2406.04136/) (2024)
- [Harnessing GPT-3.5-turbo for Rhetorical Role Prediction in Legal Cases](https://mlsanity.com/paper/2310.17413/) (2023)
- [Explainable machine learning multi-label classification of Spanish legal judgements](https://mlsanity.com/paper/2405.17610/) (2024)
- [Enhancing Pre-Trained Language Models with Sentence Position Embeddings for Rhetorical Roles Recognition in Legal Opinions](https://mlsanity.com/paper/2310.05276/) (2023)

## Built on by (most cited first, 2 of 2)

- [Segmentation and Processing of German Court Decisions from Open Legal Data](https://mlsanity.com/paper/2601.01449/) (2026)
- [Mining Legal Arguments in U.S. Corporate Case Law](https://mlsanity.com/paper/2609.25441/) (2026)

## Abstract (arXiv)

Identification of rhetorical roles like facts, arguments, and final judgments is central to understanding a legal case document and can lend power to other downstream tasks like legal case summarization and judgment prediction. However, there are several challenges to this task. Legal documents are often unstructured and contain a specialized vocabulary, making it hard for conventional transformer models to understand them. Additionally, these documents run into several pages, which makes it difficult for neural models to capture the entire context at once. Lastly, there is a dearth of annotated legal documents to train deep learning models. Previous state-of-the-art approaches for this task have focused on using neural models like BiLSTM-CRF or have explored different embedding techniques to achieve decent results. While such techniques have shown that better embedding can result in improved model performance, not many models have focused on utilizing attention for learning better embeddings in sentences of a document. Additionally, it has been recently shown that advanced techniques like multi-task learning can help the models learn better representations, thereby improving performance. In this paper, we combine these two aspects by proposing a novel family of multi-task learning-based models for rhetorical role labeling, named MARRO, that uses transformer-inspired multi-headed attention. Using label shift as an auxiliary task, we show that models from the MARRO family achieve state-of-the-art results on two labeled datasets for rhetorical role labeling, from the Indian and UK Supreme Courts.