mlsanityData to 14 Oct 2026

InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries

arXiv 2608.20220 ↗First posted 20 Aug 2026cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Legal AI systems are increasingly used to answer legal questions, yet existing benchmarks assume queries arrive fully specified. In practice, users omit facts that materially determine the legal outcome. We introduce InsufficiencyBench, the first legal benchmark targeting query-side insufficiency: whether a model recognizes when a query lacks legally material information, identifies what is missing, and refrains from premature conclusions. We formalize a taxonomy of eight canonical missing-element categories across three structural failure modes---switch, gating, and fatal prerequisite--- and construct 202 benchmark items (58 base queries, 144 deficient variants) spanning six legal domains and 24 US jurisdictions and annotated by practising attorneys. Evaluating ten frontier models, we find that no model exceeds F2 = 0.46 on missing-element identification and that the median recall is 0.44. Models either hedge indiscriminately or answer silently under fabricated presumptions. No model both identifies and qualifies responses to deficient queries while directly addressing complete ones.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 34 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

29 of 34 references.

References · 29

  1. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  2. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  3. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 336
  4. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  5. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 236
  6. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  7. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 201
  8. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 176
  9. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 150
  10. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  11. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  12. 2024LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domaincited by 118
17 more
  1. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 117
  2. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 104
  3. 2022CREPE: Open-Domain Question Answering with False Presuppositionscited by 72
  4. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  5. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 63
  6. 2024CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Modelscited by 54
  7. 2025LEXam: Benchmarking Legal Reasoning on 340 Law Examscited by 54
  8. 2025Fine-tuning Large Language Models for Improving Factuality in Legal Question Answeringcited by 28
  9. 2026There Is More to Refusal in Large Language Models than a Single Directioncited by 26
  10. 2024ClarQ-LLM: A Benchmark for Models Clarifying and Requesting Information in Task-Oriented Dialogcited by 24
  11. 2025Teaching Language Models To Gather Information Proactivelycited by 14
  12. 2026Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RLcited by 14
  13. 2025ClarifyMT-Bench: Benchmarking and Improving Multi-Turn Clarification for Conversational Large Language Modelscited by 13
  14. 2025Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Modelscited by 10
  15. 2024KG-FPQ: Evaluating Factuality Hallucination in LLMs with Knowledge Graph-based False Premise Questionscited by 9
  16. 2026Quantifying and Mitigating Premature Closure in Frontier LLMscited by 4
  17. 2025Measuring Faithfulness and Abstention: An Automated Pipeline for Evaluating LLM-Generated 3-ply Case-Based Legal Argumentscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.