# Factoring Statutory Reasoning as Language Understanding Challenges

arXiv: https://arxiv.org/abs/2105.07903 · HTML: https://mlsanity.com/paper/2105.07903/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 65 references that are in the map.
- First posted on arXiv 17 May 2021.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction](https://mlsanity.com/paper/1807.02478/) (2018)
- [Break It Down: A Question Understanding Benchmark](https://mlsanity.com/paper/2001.11770/) (2020)
- [Simple and Effective Multi-Paragraph Reading Comprehension](https://mlsanity.com/paper/1710.10723/) (2017)
- [JEC-QA: A Legal-Domain Question Answering Dataset](https://mlsanity.com/paper/1911.12011/) (2019)
- [Large-Scale Multi-Label Text Classification on EU Legislation](https://mlsanity.com/paper/1906.02192/) (2019)
- [Learning from Task Descriptions](https://mlsanity.com/paper/2011.08115/) (2020)
- [Interpretation of Natural Language Rules in Conversational Machine Reading](https://mlsanity.com/paper/1809.01494/) (2018)
- [Neural Module Networks for Reasoning over Text](https://mlsanity.com/paper/1912.04971/) (2019)
- [Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models](https://mlsanity.com/paper/2009.00751/) (2020)
- [A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering](https://mlsanity.com/paper/2005.05257/) (2020)
- [From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project](https://mlsanity.com/paper/1909.01958/) (2019)
- [Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Text](https://mlsanity.com/paper/1909.04745/) (2019)
- [Reading the Manual: Event Extraction as Definition Comprehension](https://mlsanity.com/paper/1912.01586/) (2019)
- [KERMIT: Generative Insertion-Based Modeling for Sequences](https://mlsanity.com/paper/1906.01604/) (2019)
- [Genie: A Generator of Natural Language Semantic Parsers for Virtual Assistant Commands](https://mlsanity.com/paper/1904.09020/) (2019)
- [BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understanding](https://mlsanity.com/paper/1911.00473/) (2019)
- [Procedural Reading Comprehension with Attribute-Aware Context Flow](https://mlsanity.com/paper/2003.13878/) (2020)
- [Performance in the Courtroom: Automated Processing and Visualization of Appeal Court Decisions in France](https://mlsanity.com/paper/2006.06251/) (2020)
- [DBPal: Weak Supervision for Learning a Natural Language Interface to Databases](https://mlsanity.com/paper/1909.06182/) (2019)
- [The Utility of Text: The Case of Amicus Briefs and the Supreme Court](https://mlsanity.com/paper/1409.7985/) (2014)

## Built on by (most cited first, 20 of 20)

- [Can GPT-3 Perform Statutory Reasoning?](https://mlsanity.com/paper/2302.06100/) (2023)
- [Evaluating Step-by-step Reasoning Traces: A Survey](https://mlsanity.com/paper/2502.12289/) (2025)
- [Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans](https://mlsanity.com/paper/2209.13020/) (2022)
- [Bringing order into the realm of Transformer-based language models for artificial intelligence and law](https://mlsanity.com/paper/2308.05502/) (2023)
- [Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer?](https://mlsanity.com/paper/2310.14880/) (2023)
- [The Legal Argument Reasoning Task in Civil Procedure](https://mlsanity.com/paper/2211.02950/) (2022)
- [Lawma: The Power of Specialization for Legal Annotation](https://mlsanity.com/paper/2407.16615/) (2024)
- [Automating IRAC Analysis in Malaysian Contract Law using a Semi-Structured Knowledge Base](https://mlsanity.com/paper/2406.13217/) (2024)
- [Retrieval-Augmented Chain-of-Thought in Semi-structured Domains](https://mlsanity.com/paper/2310.14435/) (2023)
- [sudoLLM: On Multi-role Alignment of Language Models](https://mlsanity.com/paper/2505.14607/) (2025)
- [Software Engineering Methods For AI-Driven Deductive Legal Reasoning](https://mlsanity.com/paper/2404.09868/) (2024)
- [Know Your Limits : On the Faithfulness of LLMs as Solvers and Autoformalizers in Legal Reasoning](https://mlsanity.com/paper/2606.16118/) (2026)
- [Policy Compliance Detection via Expression Tree Inference](https://mlsanity.com/paper/2205.12259/) (2022)
- [InSaAF: Incorporating Safety through Accuracy and Fairness | Are LLMs ready for the Indian Legal Domain?](https://mlsanity.com/paper/2402.10567/) (2024)
- [The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasure](https://mlsanity.com/paper/2507.01234/) (2025)
- [From Statute to Control Flow: Span-Grounded Deontic Trees for Defeasible Scope Parsing](https://mlsanity.com/paper/2606.08932/) (2026)
- [Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2607.14506/) (2026)
- [Towards Supporting Legal Argumentation with NLP: Is More Data Really All You Need?](https://mlsanity.com/paper/2406.10974/) (2024)
- [Investigating Multi-Agent Deliberation in Law](https://mlsanity.com/paper/2606.30906/) (2026)
- [Toward Human Rights Benchmarking for LLMs: A Pilot Methodology](https://mlsanity.com/paper/2608.10268/) (2026)

## Abstract (arXiv)

Statutory reasoning is the task of determining whether a legal statute, stated in natural language, applies to the text description of a case. Prior work introduced a resource that approached statutory reasoning as a monolithic textual entailment problem, with neural baselines performing nearly at-chance. To address this challenge, we decompose statutory reasoning into four types of language-understanding challenge problems, through the introduction of concepts and structure found in Prolog programs. Augmenting an existing benchmark, we provide annotations for the four tasks, and baselines for three of them. Models for statutory reasoning are shown to benefit from the additional structure, improving on prior baselines. Further, the decomposition into subtasks facilitates finer-grained model diagnostics and clearer incremental progress.