# Billions of Parameters Are Worth More Than In-domain Training Data: A case study in the Legal Case Entailment Task

arXiv: https://arxiv.org/abs/2205.15172 · HTML: https://mlsanity.com/paper/2205.15172/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 30 May 2022.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [A Brief Review of Domain Adaptation](https://mlsanity.com/paper/2010.03978/) (2020)
- [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://mlsanity.com/paper/2103.06268/) (2021)
- [Neural Unsupervised Domain Adaptation in NLP---A Survey](https://mlsanity.com/paper/2006.00632/) (2020)
- [Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections](https://mlsanity.com/paper/2104.04670/) (2021)
- [Language Models are Few-shot Multilingual Learners](https://mlsanity.com/paper/2109.07684/) (2021)
- [PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains](https://mlsanity.com/paper/2102.12206/) (2021)
- [Incorporating Domain Knowledge into Deep Neural Networks](https://mlsanity.com/paper/2103.00180/) (2021)
- [PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data](https://mlsanity.com/paper/2008.09144/) (2020)
- [Yes, BM25 is a Strong Baseline for Legal Case Retrieval](https://mlsanity.com/paper/2105.05686/) (2021)
- [JNLP Team: Deep Learning for Legal Processing in COLIEE 2020](https://mlsanity.com/paper/2011.08071/) (2020)
- [Rapidly Deploying a Neural Search Engine for the COVID-19 Open Research Dataset: Preliminary Thoughts and Lessons Learned](https://mlsanity.com/paper/2004.05125/) (2020)
- [Effective Approach to Develop a Sentiment Annotator For Legal Domain in a Low Resource Setting](https://mlsanity.com/paper/2011.00318/) (2020)

## Built on by (most cited first, 8 of 8)

- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [THUIR@COLIEE 2023: More Parameters and Legal Knowledge for Legal Case Entailment](https://mlsanity.com/paper/2305.06817/) (2023)
- [Efficient Distributed Optimization under Heavy-Tailed Noise](https://mlsanity.com/paper/2502.04164/) (2025)
- [Black-Box Analysis: GPTs Across Time in Legal Textual Entailment Task](https://mlsanity.com/paper/2309.05501/) (2023)
- [Simple Yet Effective Neural Ranking and Reranking Baselines for Cross-Lingual Information Retrieval](https://mlsanity.com/paper/2304.01019/) (2023)
- [NOWJ at COLIEE 2023 -- Multi-Task and Ensemble Approaches in Legal Information Processing](https://mlsanity.com/paper/2306.04903/) (2023)
- [Justifiable Artificial Intelligence: Engineering Large Language Models for Legal Applications](https://mlsanity.com/paper/2311.15716/) (2023)
- [GPTs and Language Barrier: A Cross-Lingual Legal QA Examination](https://mlsanity.com/paper/2403.18098/) (2024)

## Abstract (arXiv)

Recent work has shown that language models scaled to billions of parameters, such as GPT-3, perform remarkably well in zero-shot and few-shot scenarios. In this work, we experiment with zero-shot models in the legal case entailment task of the COLIEE 2022 competition. Our experiments show that scaling the number of parameters in a language model improves the F1 score of our previous zero-shot result by more than 6 points, suggesting that stronger zero-shot capability may be a characteristic of larger models, at least for this task. Our 3B-parameter zero-shot model outperforms all models, including ensembles, in the COLIEE 2021 test set and also achieves the best performance of a single model in the COLIEE 2022 competition, second only to the ensemble composed of the 3B model itself and a smaller version of the same model. Despite the challenges posed by large language models, mainly due to latency constraints in real-time applications, we provide a demonstration of our zero-shot monoT5-3b model being used in production as a search engine, including for legal documents. The code for our submission and the demo of our system are available at https://github.com/neuralmind-ai/coliee and https://neuralsearchx.neuralmind.ai, respectively.