# Employing Label Models on ChatGPT Answers Improves Legal Text Entailment Performance

arXiv: https://arxiv.org/abs/2401.17897 · HTML: https://mlsanity.com/paper/2401.17897/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 26 references that are in the map.
- First posted on arXiv 31 Jan 2024.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods](https://mlsanity.com/paper/2002.11955/) (2020)
- [A Survey on Programmatic Weak Supervision](https://mlsanity.com/paper/2202.05433/) (2022)
- [When do you need Chain-of-Thought Prompting for ChatGPT?](https://mlsanity.com/paper/2304.03262/) (2023)
- [Learning Hyper Label Model for Programmatic Weak Supervision](https://mlsanity.com/paper/2207.13545/) (2022)
- [JNLP Team: Deep Learning for Legal Processing in COLIEE 2020](https://mlsanity.com/paper/2011.08071/) (2020)
- [Leveraging Instance Features for Label Aggregation in Programmatic Weak Supervision](https://mlsanity.com/paper/2210.02724/) (2022)
- [ParaLaw Nets -- Cross-lingual Sentence-level Pretraining for Legal Text Processing](https://mlsanity.com/paper/2106.13403/) (2021)

## Built on by (most cited first, 2 of 2)

- [Large Language Models Meet Legal Artificial Intelligence: A Survey](https://mlsanity.com/paper/2509.09969/) (2025)
- [Exploring the Nexus of Large Language Models and Legal Systems: A Short Survey](https://mlsanity.com/paper/2404.00990/) (2024)

## Abstract (arXiv)

The objective of legal text entailment is to ascertain whether the assertions in a legal query logically follow from the information provided in one or multiple legal articles. ChatGPT, a large language model, is robust in many natural language processing tasks, including legal text entailment: when we set the temperature = 0 (the ChatGPT answers are deterministic) and prompt the model, it achieves 70.64% accuracy on COLIEE 2022 dataset, which outperforms the previous SOTA of 67.89%. On the other hand, if the temperature is larger than zero, ChatGPT answers are not deterministic, leading to inconsistent answers and fluctuating results. We propose to leverage label models (a fundamental component of weak supervision techniques) to integrate the provisional answers by ChatGPT into consolidated labels. By that way, we treat ChatGPT provisional answers as noisy predictions which can be consolidated by label models. The experimental results demonstrate that this approach can attain an accuracy of 76.15%, marking a significant improvement of 8.26% over the prior state-of-the-art benchmark. Additionally, we perform an analysis of the instances where ChatGPT produces incorrect answers, then we classify the errors, offering insights that could guide potential enhancements for future research endeavors.