# Understand Legal Documents with Contextualized Large Language Models

arXiv: https://arxiv.org/abs/2303.12135 · HTML: https://mlsanity.com/paper/2303.12135/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 21 Mar 2023.

## Builds on (most cited first, 23 of 23)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [A Survey on Deep Learning for Named Entity Recognition](https://mlsanity.com/paper/1812.09449/) (2018)
- [A Critical Review of Recurrent Neural Networks for Sequence Learning](https://mlsanity.com/paper/1506.00019/) (2015)
- [Pretrained Language Models for Sequential Sentence Classification](https://mlsanity.com/paper/1909.04054/) (2019)
- [Identification of Rhetorical Roles of Sentences in Indian Legal Judgments](https://mlsanity.com/paper/1911.05405/) (2019)
- [Semantic Segmentation of Legal Documents via Rhetorical Roles](https://mlsanity.com/paper/2112.01836/) (2021)
- [Named Entity Recognition in Indian court judgments](https://mlsanity.com/paper/2211.03442/) (2022)
- [NRTSI: Non-Recurrent Time Series Imputation](https://mlsanity.com/paper/2102.03340/) (2021)
- [Corpus for Automatic Structuring of Legal Documents](https://mlsanity.com/paper/2201.13125/) (2022)
- [Hierarchical Neural Networks for Sequential Sentence Classification in Medical Scientific Abstracts](https://mlsanity.com/paper/1808.06161/) (2018)
- [On Sampling Top-K Recommendation Evaluation](https://mlsanity.com/paper/2106.10621/) (2021)
- [mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2110.08151/) (2021)
- [Cross-Domain Multi-Task Learning for Sequential Sentence Classification in Research Papers](https://mlsanity.com/paper/2102.06008/) (2021)
- [On Estimating Recommendation Evaluation Metrics under Sampling](https://mlsanity.com/paper/2103.01474/) (2021)
- [End-to-End Subtitle Detection and Recognition for Videos in East Asian Languages via CNN Ensemble with Near-Human-Level Performance](https://mlsanity.com/paper/1611.06159/) (2016)
- [Transparent Single-Cell Set Classification with Kernel Mean Embeddings](https://mlsanity.com/paper/2201.07322/) (2022)
- [Distribution-based Sketching of Single-Cell Samples](https://mlsanity.com/paper/2207.00584/) (2022)
- [Large-scale Real-time Personalized Similar Product Recommendations](https://mlsanity.com/paper/2004.05716/) (2020)

## Built on by (most cited first, 5 of 5)

- [Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Review](https://mlsanity.com/paper/2402.10350/) (2024)
- [TaskCLIP: Extend Large Vision-Language Model for Task Oriented Object Detection](https://mlsanity.com/paper/2403.08108/) (2024)
- [CloudEval-YAML: A Practical Benchmark for Cloud Configuration Generation](https://mlsanity.com/paper/2401.06786/) (2023)
- [Comprehensive evaluation of Mal-API-2019 dataset by machine learning in malware detection](https://mlsanity.com/paper/2403.02232/) (2024)
- [Can Contrastive Learning Refine Embeddings](https://mlsanity.com/paper/2404.08701/) (2024)

## Abstract (arXiv)

The growth of pending legal cases in populous countries, such as India, has become a major issue. Developing effective techniques to process and understand legal documents is extremely useful in resolving this problem. In this paper, we present our systems for SemEval-2023 Task 6: understanding legal texts (Modi et al., 2023). Specifically, we first develop the Legal-BERT-HSLN model that considers the comprehensive context information in both intra- and inter-sentence levels to predict rhetorical roles (subtask A) and then train a Legal-LUKE model, which is legal-contextualized and entity-aware, to recognize legal entities (subtask B). Our evaluations demonstrate that our designed models are more accurate than baselines, e.g., with an up to 15.0% better F1 score in subtask B. We achieved notable performance in the task leaderboard, e.g., 0.834 micro F1 score, and ranked No.5 out of 27 teams in subtask A.