# An Evaluation Framework for Legal Document Summarization

arXiv: https://arxiv.org/abs/2205.08478 · HTML: https://mlsanity.com/paper/2205.08478/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 17 May 2022.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [A Survey of Evaluation Metrics Used for NLG Systems](https://mlsanity.com/paper/2008.12009/) (2020)
- [Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting](https://mlsanity.com/paper/1805.11080/) (2018)
- [Leveraging BERT for Extractive Text Summarization on Lectures](https://mlsanity.com/paper/1906.04165/) (2019)
- [Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation](https://mlsanity.com/paper/1805.11004/) (2018)
- [Extractive Summarization using Deep Learning](https://mlsanity.com/paper/1708.04439/) (2017)

## Built on by (most cited first, 10 of 10)

- [Text Takes Over: A Study of Modality Bias in Multimodal Intent Detection](https://mlsanity.com/paper/2508.16122/) (2025)
- [A Comprehensive Survey on Legal Summarization: Challenges and Future Directions](https://mlsanity.com/paper/2501.17830/) (2025)
- [On The Persona-based Summarization of Domain-Specific Documents](https://mlsanity.com/paper/2406.03986/) (2024)
- [A Pointer Network-based Approach for Joint Extraction and Detection of Multi-Label Multi-Class Intents](https://mlsanity.com/paper/2410.22476/) (2024)
- [Introducing Spotlight: A Novel Approach for Generating Captivating Key Information from Documents](https://mlsanity.com/paper/2509.10935/) (2025)
- [Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/2601.15267/) (2026)
- [Towards Supporting Legal Argumentation with NLP: Is More Data Really All You Need?](https://mlsanity.com/paper/2406.10974/) (2024)
- [PL-CA: A Parametric Legal Case Augmentation Framework](https://mlsanity.com/paper/2509.06356/) (2025)
- [Na Pr'atica, qual IA Entende o Direito? Um Estudo Experimental com IAs Generalistas e uma IA Jur'idica](https://mlsanity.com/paper/2510.18108/) (2025)
- [IDALC: A Semi-Supervised Framework for Intent Detection and Active Learning based Correction](https://mlsanity.com/paper/2511.05921/) (2025)

## Abstract (arXiv)

A law practitioner has to go through numerous lengthy legal case proceedings for their practices of various categories, such as land dispute, corruption, etc. Hence, it is important to summarize these documents, and ensure that summaries contain phrases with intent matching the category of the case. To the best of our knowledge, there is no evaluation metric that evaluates a summary based on its intent. We propose an automated intent-based summarization metric, which shows a better agreement with human evaluation as compared to other automated metrics like BLEU, ROUGE-L etc. in terms of human satisfaction. We also curate a dataset by annotating intent phrases in legal documents, and show a proof of concept as to how this system can be automated. Additionally, all the code and data to generate reproducible results is available on Github.