mlsanityData to 7 Oct 2026

Can Legal AI Know When It Is Wrong? And Do Students Know When It Is?

arXiv 2608.21089 ↗First posted 21 Aug 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Integrating Large Language Models (LLMs) into the Indian judiciary promises access to justice but introduces severe risks. We identify the 'inertia of confidence'--an overconfidence phenomenon analogous to the Dunning-Kruger effect where LLMs provide incorrect legal verdicts with near-maximum confidence, driven by a hypothesized 'precedent overfitting' bias. Phase I of our socio-technical audit tested ChatGPT (GPT-5.2), Meta AI, and Perplexity AI on a 60-case battery regarding the Indian Contract Act, 1872, and the shift toward statutory enforcement of specific performance. We introduce the High-Confidence Error Rate (HCER) to quantify incorrect verdicts delivered with dangerous certainty (>= 9 on a 1-10 scale). All models struggled with statutory updates. Meta AI proved most vulnerable (31.7% HCER), frequently misapplying pre-amendment rules with a 9.1/10 mean confidence, followed by Perplexity (15.0%) and ChatGPT (6.7%). Phase II investigated human vulnerability to this overconfidence via a survey of Indian law students (N=380). Verification often functions as a reactive adaptation to machine hallucinations: students encountering fabricated citations reported higher verification scores (4.2/5) than those with no such encounters (2.8/5). Furthermore, while 81.6% knew submitting hallucinated cases can lead to contempt-of-court, 71.1% received no formal training on ethical AI use. We propose shifting toward adversarial legal research pedagogy and implementing source-grounded verification architectures to prevent systemic professional negligence.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 11 of its 32 references that are in the map. First posted on arXiv 21 Aug 2026.

Builds on

11 of 32 references.

References · 11

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  3. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  4. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  5. 2022Pre-trained Language Models for the Legal Domain: A Case Study on Indian Lawcited by 39
  6. 2024Performance and Metacognition Disconnect when Reasoning in Human-AI Interactioncited by 10
  7. 2025BhashaBench V1: A Comprehensive Benchmark for the Quadrant of Indic Domainscited by 6
  8. 2021Indian Legal NLP Benchmarks : A Surveycited by 3
  9. 2025Navigating AI Policy Landscapes: Insights into Human Rights Considerations Across IEEE Regionscited by 2
  10. 2025Are LLMs Court-Ready? Evaluating Frontier Models on Indian Legal Reasoningcited by 2
  11. 2025Ethical Challenges of Using Artificial Intelligence in Judiciarycited by 1

Built on by

Not cited yet.