# A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering

arXiv: https://arxiv.org/abs/2005.05257 · HTML: https://mlsanity.com/paper/2005.05257/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2020.

## Built on by (most cited first, 30 of 62)

- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://mlsanity.com/paper/2103.06268/) (2021)
- [ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts](https://mlsanity.com/paper/2110.01799/) (2021)
- [Can GPT-3 Perform Statutory Reasoning?](https://mlsanity.com/paper/2302.06100/) (2023)
- [Legal Prompt Engineering for Multilingual Legal Judgement Prediction](https://mlsanity.com/paper/2212.02199/) (2022)
- ["According to ...": Prompting Language Models Improves Quoting from Pre-Training Data](https://mlsanity.com/paper/2305.13252/) (2023)
- [A Statutory Article Retrieval Dataset in French](https://mlsanity.com/paper/2108.11792/) (2021)
- [A Reasoning-Focused Legal Retrieval Benchmark](https://mlsanity.com/paper/2505.03970/) (2025)
- [Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2309.17050/) (2023)
- [A Survey of Useful LLM Evaluation](https://mlsanity.com/paper/2406.00936/) (2024)
- [Flexible Generation of Natural Language Deductions](https://mlsanity.com/paper/2104.08825/) (2021)
- [Factoring Statutory Reasoning as Language Understanding Challenges](https://mlsanity.com/paper/2105.07903/) (2021)
- [Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack](https://mlsanity.com/paper/2407.16695/) (2024)
- [CLERC: A Dataset for Legal Case Retrieval and Retrieval-Augmented Analysis Generation](https://mlsanity.com/paper/2406.17186/) (2024)
- [Zero-shot Transfer of Article-aware Legal Outcome Classification for European Court of Human Rights Cases](https://mlsanity.com/paper/2302.00609/) (2023)
- [VeriCoT: Neuro-symbolic Chain-of-Thought Validation via Logical Consistency Checks](https://mlsanity.com/paper/2511.04662/) (2025)
- [A Survey on Legal Question Answering Systems](https://mlsanity.com/paper/2110.07333/) (2021)
- [BLT: Can Large Language Models Handle Basic Legal Text?](https://mlsanity.com/paper/2311.09693/) (2023)
- [LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domain](https://mlsanity.com/paper/2404.02127/) (2024)
- [Semi-Supervised Text Classification via Self-Pretraining](https://mlsanity.com/paper/2109.15300/) (2021)
- [The Legal Argument Reasoning Task in Civil Procedure](https://mlsanity.com/paper/2211.02950/) (2022)
- [Faithful Autoformalization via Roundtrip Verification and Repair](https://mlsanity.com/paper/2604.25031/) (2026)
- [Cross-Policy Compliance Detection via Question Answering](https://mlsanity.com/paper/2109.03731/) (2021)
- [Automating IRAC Analysis in Malaysian Contract Law using a Semi-Structured Knowledge Base](https://mlsanity.com/paper/2406.13217/) (2024)
- [JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment](https://mlsanity.com/paper/2605.25240/) (2026)
- [Evaluating Document Representations for Content-based Legal Literature Recommendations](https://mlsanity.com/paper/2104.13841/) (2021)
- [No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inference](https://mlsanity.com/paper/2306.09918/) (2023)
- [Multiple-Source Domain Adaptation via Coordinated Domain Encoders and Paired Classifiers](https://mlsanity.com/paper/2201.11870/) (2022)
- [Creation and Analysis of an International Corpus of Privacy Laws](https://mlsanity.com/paper/2206.14169/) (2022)
- [Software Engineering Methods For AI-Driven Deductive Legal Reasoning](https://mlsanity.com/paper/2404.09868/) (2024)

## Abstract (arXiv)

Legislation can be viewed as a body of prescriptive rules expressed in natural language. The application of legislation to facts of a case we refer to as statutory reasoning, where those facts are also expressed in natural language. Computational statutory reasoning is distinct from most existing work in machine reading, in that much of the information needed for deciding a case is declared exactly once (a law), while the information needed in much of machine reading tends to be learned through distributional language statistics. To investigate the performance of natural language understanding approaches on statutory reasoning, we introduce a dataset, together with a legal-domain text corpus. Straightforward application of machine reading models exhibits low out-of-the-box performance on our questions, whether or not they have been fine-tuned to the legal domain. We contrast this with a hand-constructed Prolog-based system, designed to fully solve the task. These experiments support a discussion of the challenges facing statutory reasoning moving forward, which we argue is an interesting real-world task that can motivate the development of models able to utilize prescriptive rules specified in natural language.