# Challenges for Generative AI in Legal Reasoning arXiv: https://arxiv.org/abs/2508.18880 ยท HTML: https://mlsanity.com/paper/2508.18880/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. - Builds on 19 of its 50 references that are in the map. - First posted on arXiv 26 Aug 2025. ## Placed in - LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/) ## Builds on (most cited first, 19 of 19) - [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023) - [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024) - [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024) - [Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models](https://mlsanity.com/paper/2401.01301/) (2024) - [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023) - [LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain](https://mlsanity.com/paper/2408.10343/) (2024) - [Factuality of Large Language Models: A Survey](https://mlsanity.com/paper/2402.02420/) (2024) - [The Perils & Promises of Fact-checking with Large Language Models](https://mlsanity.com/paper/2310.13549/) (2023) - [Towards Robust Legal Reasoning: Harnessing Logical LLMs in Law](https://mlsanity.com/paper/2502.17638/) (2025) - [Fine-tuning Large Language Models for Improving Factuality in Legal Question Answering](https://mlsanity.com/paper/2501.06521/) (2025) - [Strong and weak alignment of large language models with human values](https://mlsanity.com/paper/2408.04655/) (2024) - [More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLMs](https://mlsanity.com/paper/2405.17830/) (2024) - [SteerConf: Steering LLMs for Confidence Elicitation](https://mlsanity.com/paper/2503.02863/) (2025) - [Can Large Language Models Grasp Legal Theories? Enhance Legal Reasoning with Insights from Multi-Agent Collaboration](https://mlsanity.com/paper/2410.02507/) (2024) - [Generative Interpretation](https://mlsanity.com/paper/2308.06907/) (2023) - [Evaluating Evidence Attribution in Generated Fact Checking Explanations](https://mlsanity.com/paper/2406.12645/) (2024) - [Incorporating Legal Structure in Retrieval-Augmented Generation: A Case Study on Copyright Fair Use](https://mlsanity.com/paper/2505.02164/) (2025) - [TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputs](https://mlsanity.com/paper/2507.08203/) (2025) - [Adversarial Negotiation Dynamics in Generative Language Models](https://mlsanity.com/paper/2501.00069/) (2024) ## Built on by (most cited first, 7 of 7) - [Adaptive Collaboration of Arena-Based Argumentative LLMs for Explainable and Contestable Legal Reasoning](https://mlsanity.com/paper/2602.18916/) (2026) - [Benchmarked Yet Not Measured -- Generative AI Should be Evaluated Against Real-World Utility](https://mlsanity.com/paper/2605.06856/) (2026) - [Which Changes Matter? Towards Trustworthy Legal AI via Relevance-Sensitive Evaluation and Solver-Grounded Reasoning](https://mlsanity.com/paper/2605.26530/) (2026) - [Beyond Probabilistic Similarity: Structural, Temporal, and Causal Limitations of Retrieval-Augmented Generation in the Legal Domain](https://mlsanity.com/paper/2606.09724/) (2026) - [Can We Trust LLM's Logic? Quantifying Uncertainty, Coherence, and Robustness via a Graph-Based Framework](https://mlsanity.com/paper/2607.08017/) (2026) - [LexAgentHallu: A Hierarchical Benchmark for Profiling Hallucinations in Legal Agents](https://mlsanity.com/paper/2609.09754/) (2026) - [JusticeAxis: Benchmarking Legal Judgment between Rigid Rule Application and Ungrounded Discretion](https://mlsanity.com/paper/2610.00353/) (2026) ## Abstract (arXiv) Large Language Models (LLMs) are being integrated into professional domains, yet their limitations in such high-stakes fields as law remain poorly understood. In response, this paper introduces examples of critical challenges to the functioning of generative and other forms of artificial intelligence (AI) as reliable reasoning tools in judicial decision-making. The study deconstructs core requirements and challenges for AI, including the ability to select the correct legal framework across jurisdictions, generate sound arguments based on the doctrine of the sources of law, distinguish ratio decidendi and obiter dicta in case law, resolve ambiguity arising from general clauses like "reasonableness", manage conflicting legal provisions, and apply the burden of proof correctly. The paper maps various AI enhancement mechanisms, such as retrieval-augmented generation (RAG), multi-agent systems and neuro-symbolic AI, to these challenges, assessing their potential to bridge the gap between the probabilistic nature of LLMs and the rigorous, choice-driven demands of legal interpretation. Furthermore, the paper sketches a path towards an evaluation framework, proposing that legal requirements be organized into normative, doctrinal, evidential, and technical categories, and subsequently operationalized into domain-specific, testable design obligations. The findings indicate that these techniques can address specific narrow challenges, but they fail to solve the more significant ones, particularly in tasks requiring discretion and transparent, justifiable reasoning. Therefore, we advocate for a staged adoption, first capturing efficiency in simple cases with technology already available today and sustaining long-term investment in new methods that handle hierarchy, temporality, and other requirements of legally sound reasoning, thus enabling expansion to complex adjudication in the future.