# ParaLaw Nets -- Cross-lingual Sentence-level Pretraining for Legal Text Processing

arXiv: https://arxiv.org/abs/2106.13403 · HTML: https://mlsanity.com/paper/2106.13403/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 16 references that are in the map.
- First posted on arXiv 25 Jun 2021.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [JNLP Team: Deep Learning for Legal Processing in COLIEE 2020](https://mlsanity.com/paper/2011.08071/) (2020)

## Built on by (most cited first, 5 of 5)

- [Attentive Deep Neural Networks for Legal Document Retrieval](https://mlsanity.com/paper/2212.13899/) (2022)
- [A Summary of the ALQAC 2021 Competition](https://mlsanity.com/paper/2204.10717/) (2022)
- [CAPTAIN at COLIEE 2023: Efficient Methods for Legal Information Retrieval and Entailment Tasks](https://mlsanity.com/paper/2401.03551/) (2024)
- [Transformer-based Approaches for Legal Text Processing](https://mlsanity.com/paper/2202.06397/) (2022)
- [Employing Label Models on ChatGPT Answers Improves Legal Text Entailment Performance](https://mlsanity.com/paper/2401.17897/) (2024)

## Abstract (arXiv)

Ambiguity is a characteristic of natural language, which makes expression ideas flexible. However, in a domain that requires accurate statements, it becomes a barrier. Specifically, a single word can have many meanings and multiple words can have the same meaning. When translating a text into a foreign language, the translator needs to determine the exact meaning of each element in the original sentence to produce the correct translation sentence. From that observation, in this paper, we propose ParaLaw Nets, a pretrained model family using sentence-level cross-lingual information to reduce ambiguity and increase the performance in legal text processing. This approach achieved the best result in the Question Answering task of COLIEE-2021.