mlsanityData to 7 Oct 2026

Which Changes Matter? Towards Trustworthy Legal AI via Relevance-Sensitive Evaluation and Solver-Grounded Reasoning

arXiv 2605.26530 ↗First posted 26 May 2026cs.AIcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Legal reasoning requires distinguishing changes that matter from those that do not. Legal AI should remain stable under legally irrelevant perturbations, but should change when perturbations alter legally material points. We formulate this requirement as a legal-relevance-sensitive evaluation problem: LLMs should only be sensitive to the legally relevant change. We introduce a unified evaluation suite covering should-change and should-not-change evaluation across judicial fairness, robustness, and statute-confusion scenarios. Our evaluation shows that existing legal LLMs are systematically sensitive to legally irrelevant variations and often fail to distinguish related legal elements and statutory rules. To mitigate these failures, we present LexGuard, an adversarial multi-agent framework grounded in formal reasoning. LexGuard formalizes statutes into executable constraints, uses adversarial agents to extract competing fact-statute arguments, and invokes SMT solvers to verify legal satisfaction and logical consistency. Experiments show that LexGuard improves legal reasoning reliability by reducing vulnerability to manipulative framing, improving disambiguation among similar statutes, limiting the influence of legally irrelevant attributes, and increasing consistency under benign reformulations. We show that legal trustworthiness requires not only accuracy, but calibrated sensitivity to legally material changes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 10 of its 32 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

10 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Challenges for Generative AI in Legal Reasoning26 Aug
  2. 2025Explainable Rule Application via Structured Prompting: A Neural-Symbolic Approach19 Jun
  3. 2025Towards Robust Legal Reasoning: Harnessing Logical LLMs in Law24 Feb

Other references · 7

  1. 2023Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecturecited by 119
  2. 2023DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Servicescited by 117
  3. 2023LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Datasetcited by 37
  4. 2024AgentCourt: Simulating Court with Adversarial Evolvable Lawyer Agentscited by 36
  5. 2024Agents on the Bench: Large Language Model Based Multi Agent Framework for Trustworthy Digital Justicecited by 10
  6. 2025Incorporating Legal Structure in Retrieval-Augmented Generation: A Case Study on Copyright Fair Usecited by 6
  7. 2024Gaps or Hallucinations? Gazing into Machine-Generated Legal Analysis for Fine-grained Text Evaluationscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Trustworthy Legal AI through LLM Agents and Formal Reasoning
  2. 2026Legal Reasoning Is Not Lawyering: Rethinking Legal Benchmarks for Pro Se Access to Justice
  3. 2026Bridging Legal Interpretation and Formal Logic: Faithfulness, Assumption, and the Future of AI Legal Reasoning
  4. 2026Closing the Loop: Formally Verified Law as a Reward Signal for Self-Improving Legal AI
  5. 2025Better Call CLAUSE: A Discrepancy Benchmark for Auditing LLMs Legal Reasoning Capabilities
  6. 2025Debate-Feedback: A Multi-Agent Framework for Efficient Legal Judgment Prediction
  7. 2026The Measurement Gap in the Automation of EU Law: Benchmarking Doctrinal Legal Reasoning under the EU AI Act
  8. 2026LRAS: Advanced Legal Reasoning with Agentic Search