mlsanityData to 7 Oct 2026

LLMs Provide Unstable Answers to Legal Questions

arXiv 2502.05196 ↗First posted 28 Jan 2025cs.CL cs.CYcited by 19

Not on the map. Why?

Abstract (arXiv)

An LLM is stable if it reaches the same conclusion when asked the identical question multiple times. We find leading LLMs like gpt-4o, claude-3.5, and gemini-1.5 are unstable when providing answers to hard legal questions, even when made as deterministic as possible by setting temperature to 0. We curate and release a novel dataset of 500 legal questions distilled from real cases, involving two parties, with facts, competing legal arguments, and the question of which party should prevail. When provided the exact same question, we observe that LLMs sometimes say one party should win, while other times saying the other party should win. This instability has implications for the increasing numbers of legal AI products, legal processes, and lawyers relying on these LLMs.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 5 of its 20 references that are in the map. First posted on arXiv 28 Jan 2025.

Builds on

5 of 20 references.

References · 5

  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2024The Effect of Sampling Temperature on Problem Solving in Large Language Modelscited by 168
  3. 2023Large Language Models in Law: A Surveycited by 128
  4. 2024Are Your LLMs Capable of Stable Reasoning?cited by 65
  5. 2024Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scorescited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 9.

19 papers.

Sort:

By area:

  1. 2025Towards Robust Legal Reasoning: Harnessing Logical LLMs in Lawcited by 30
  2. 2025On Verifiable Legal Reasoning: A Multi-Agent Framework with Formalized Knowledge Representationscited by 16
  3. 2025Explainable Rule Application via Structured Prompting: A Neural-Symbolic Approachcited by 14
  4. 2025Unilaw-R1: A Large Language Model for Legal Reasoning with Reinforcement Learning and Iterative Inferencecited by 10
  5. 2026LRAS: Advanced Legal Reasoning with Agentic Searchcited by 8
  6. 2025Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variabilitycited by 6
  7. 2025Evaluating LLM-Generated Legal Explanations for Regulatory Compliance in Social Media Influencer Marketingcited by 4
  8. 2025Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysiscited by 4
  9. 2026MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Gamescited by 4
  10. 2025Bye-bye, Bluebook? Automating Legal Drudgery With AI-Augmented Rule Followingcited by 3
  11. 2026LawThinker: A Deep Research Legal Agent in Dynamic Environmentscited by 3
  12. 2025MetaCipher: A Time-Persistent and Universal Multi-Agent Framework for Cipher-Based Jailbreak Attacks for LLMscited by 1
7 more
  1. 2026Structured Decomposition for LLM Reasoning: Cross-Domain Validation and Semantic Web Integrationcited by 1
  2. 2026Communication Enhances LLMs' Stability in Strategic Thinkingcited by 1
  3. 2026Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coveragecited by 1
  4. 2025Evidence of conceptual mastery in the application of rules by Large Language Modelscited by 0
  5. 2026Assessing and Explaining the Persuadability of Large Language Models as Legal Decision Toolscited by 0
  6. 2026From Judgments to Issues: Structured Extraction of Legal Reasoning with Citation-Hallucination Controlcited by 0
  7. 2026Cheap Talk Stabilizes Strategic Interaction in LLM Agentscited by 0