# Transformer-based Approaches for Legal Text Processing

arXiv: https://arxiv.org/abs/2202.06397 · HTML: https://mlsanity.com/paper/2202.06397/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 13 Feb 2022.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Building Legal Case Retrieval Systems with Lexical Matching and Summarization using A Pre-Trained Phrase Scoring Model](https://mlsanity.com/paper/2009.14083/) (2020)
- [JNLP Team: Deep Learning for Legal Processing in COLIEE 2020](https://mlsanity.com/paper/2011.08071/) (2020)
- [Legal Document Retrieval using Document Vector Embeddings and Deep Learning](https://mlsanity.com/paper/1805.10685/) (2018)
- [IITP@COLIEE 2019: Legal Information Retrieval using BM25 and BERT](https://mlsanity.com/paper/2104.08653/) (2021)
- [ParaLaw Nets -- Cross-lingual Sentence-level Pretraining for Legal Text Processing](https://mlsanity.com/paper/2106.13403/) (2021)
- [Threshold-Based Retrieval and Textual Entailment Detection on Legal Bar Exam Questions](https://mlsanity.com/paper/1905.13350/) (2019)

## Built on by (most cited first, 3 of 3)

- [Black-Box Analysis: GPTs Across Time in Legal Textual Entailment Task](https://mlsanity.com/paper/2309.05501/) (2023)
- [RMDM: A Multilabel Fakenews Dataset for Vietnamese Evidence Verification](https://mlsanity.com/paper/2309.09071/) (2023)
- [LBMT team at VLSP2022-Abmusu: Hybrid method with text correlation and generative models for Vietnamese multi-document summarization](https://mlsanity.com/paper/2304.05205/) (2023)

## Abstract (arXiv)

In this paper, we introduce our approaches using Transformer-based models for different problems of the COLIEE 2021 automatic legal text processing competition. Automated processing of legal documents is a challenging task because of the characteristics of legal documents as well as the limitation of the amount of data. With our detailed experiments, we found that Transformer-based pretrained language models can perform well with automated legal text processing problems with appropriate approaches. We describe in detail the processing steps for each task such as problem formulation, data processing and augmentation, pretraining, finetuning. In addition, we introduce to the community two pretrained models that take advantage of parallel translations in legal domain, NFSP and NMSP. In which, NFSP achieves the state-of-the-art result in Task 5 of the competition. Although the paper focuses on technical reporting, the novelty of its approaches can also be an useful reference in automated legal document processing using Transformer-based models.