mlsanityData to 7 Oct 2026

A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering

arXiv 2005.05257 ↗First posted 11 May 2020cs.CLcited by 61

Not on the map. Why?

Abstract (arXiv)

Legislation can be viewed as a body of prescriptive rules expressed in natural language. The application of legislation to facts of a case we refer to as statutory reasoning, where those facts are also expressed in natural language. Computational statutory reasoning is distinct from most existing work in machine reading, in that much of the information needed for deciding a case is declared exactly once (a law), while the information needed in much of machine reading tends to be learned through distributional language statistics. To investigate the performance of natural language understanding approaches on statutory reasoning, we introduce a dataset, together with a legal-domain text corpus. Straightforward application of machine reading models exhibits low out-of-the-box performance on our questions, whether or not they have been fine-tuned to the legal domain. We contrast this with a hand-constructed Prolog-based system, designed to fully solve the task. These experiments support a discussion of the challenges facing statutory reasoning moving forward, which we argue is an interesting real-world task that can motivate the development of models able to utilize prescriptive rules specified in natural language.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 6; 2023: 8; 2024: 9; 2025: 7; 2026: 22.

61 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  3. 2021ContractNLI: A Dataset for Document-level Natural Language Inference for Contractscited by 145
  4. 2023Can GPT-3 Perform Statutory Reasoning?cited by 71
  5. 2022Legal Prompt Engineering for Multilingual Legal Judgement Predictioncited by 54
  6. 2023"According to ...": Prompting Language Models Improves Quoting from Pre-Training Datacited by 42
  7. 2021A Statutory Article Retrieval Dataset in Frenchcited by 40
  8. 2025A Reasoning-Focused Legal Retrieval Benchmarkcited by 37
  9. 2023Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Modelscited by 35
  10. 2021Flexible Generation of Natural Language Deductionscited by 28
  11. 2024A Survey of Useful LLM Evaluationcited by 28
  12. 2021Factoring Statutory Reasoning as Language Understanding Challengescited by 20
38 more
  1. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  2. 2024CLERC: A Dataset for Legal Case Retrieval and Retrieval-Augmented Analysis Generationcited by 18
  3. 2023Zero-shot Transfer of Article-aware Legal Outcome Classification for European Court of Human Rights Casescited by 14
  4. 2021A Survey on Legal Question Answering Systemscited by 10
  5. 2023BLT: Can Large Language Models Handle Basic Legal Text?cited by 10
  6. 2024LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domaincited by 10
  7. 2025VeriCoT: Neuro-symbolic Chain-of-Thought Validation via Logical Consistency Checkscited by 10
  8. 2021Semi-Supervised Text Classification via Self-Pretrainingcited by 8
  9. 2022The Legal Argument Reasoning Task in Civil Procedurecited by 8
  10. 2021Cross-Policy Compliance Detection via Question Answeringcited by 7
  11. 2026Faithful Autoformalization via Roundtrip Verification and Repaircited by 6
  12. 2024Automating IRAC Analysis in Malaysian Contract Law using a Semi-Structured Knowledge Basecited by 5
  13. 2021Evaluating Document Representations for Content-based Legal Literature Recommendationscited by 4
  14. 2023No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inferencecited by 4
  15. 2022Multiple-Source Domain Adaptation via Coordinated Domain Encoders and Paired Classifierscited by 3
  16. 2022Creation and Analysis of an International Corpus of Privacy Lawscited by 3
  17. 2024Software Engineering Methods For AI-Driven Deductive Legal Reasoningcited by 3
  18. 2025Taxation Perspectives from Large Language Models: A Case Study on Additional Tax Penaltiescited by 3
  19. 2026OpenExempt: A Diagnostic Benchmark for Legal Reasoning and a Framework for Creating Custom Benchmarks on Demandcited by 3
  20. 2026JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessmentcited by 3
  21. 2026Know Your Limits : On the Faithfulness of LLMs as Solvers and Autoformalizers in Legal Reasoningcited by 3
  22. 2023NOWJ1@ALQAC 2023: Enhancing Legal Task Performance with Classic Statistical Models and Pre-trained Language Modelscited by 2
  23. 2022Policy Compliance Detection via Expression Tree Inferencecited by 1
  24. 2022Validity Assessment of Legal Will Statements as Natural Language Inferencecited by 1
  25. 2024Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Studycited by 1
  26. 2024A Comprehensive Framework for Reliable Legal AI: Combining Specialized Expert Systems and Adaptive Refinementcited by 1
  27. 2025LegalViz: Legal Text Visualization by Text To Diagram Generationcited by 1
  28. 2026Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directionscited by 1
  29. 2026Where Experts Disagree, Models Fail: Detecting Implicit Legal Citations in French Court Decisionscited by 1
  30. 2026From Statute to Control Flow: Span-Grounded Deontic Trees for Defeasible Scope Parsingcited by 1
  31. 2026Reasoning Consensus: Structural Ensembling of LLM Reasoning via Weighted DAG Aggregationcited by 1
  32. 2024A Method for Detecting Legal Article Competition for Korean Criminal Law Using a Case-augmented Mention Graphcited by 0
  33. 2025Transformer-Based Extraction of Statutory Definitions from the U.S. Codecited by 0
  34. 2025LLM-Assisted Formalization Enables Deterministic Detection of Statutory Inconsistency in the Internal Revenue Codecited by 0
  35. 2025Streamlining Industrial Contract Management with Retrieval-Augmented LLMscited by 0
  36. 2026AI-Assisted Moot Courts: Simulating Justice-Specific Questioning in Oral Argumentscited by 0
  37. 2026Learning When Not to Decide: A Framework for Overcoming Factual Presumptuousness in AI Adjudicationcited by 0
  38. 2026Reasoners or Translators? Contamination-aware Evaluation and Neuro-Symbolic Robustness in Tax Lawcited by 0

50 most cited of 61 All 61 (JSON)