mlsanityData to 14 Oct 2026

Are LLMs Court-Ready? Evaluating Frontier Models on Indian Legal Reasoning

arXiv 2510.17900 ↗First posted 19 Oct 2025cs.CY cs.AI cs.CLcited by 2

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Large language models (LLMs) are entering legal workflows, yet we lack a jurisdiction-specific framework to assess their baseline competence therein. We use India's public legal examinations as a transparent proxy. Our multi-year benchmark assembles objective screens from top national and state exams and evaluates open and frontier LLMs under real-world exam conditions. To probe beyond multiple-choice questions, we also include a lawyer-graded, paired-blinded study of long-form answers from the Supreme Court's Advocate-on-Record exam. This is, to our knowledge, the first exam-grounded, India-specific yardstick for LLM court-readiness released with datasets and protocols. Our work shows that while frontier systems consistently clear historical cutoffs and often match or exceed recent top-scorer bands on objective exams, none surpasses the human topper on long-form reasoning. Grader notes converge on three reliability failure modes: procedural or format compliance, authority or citation discipline, and forum-appropriate voice and structure. These findings delineate where LLMs can assist (checks, cross-statute consistency, statute and precedent lookups) and where human leadership remains essential: forum-specific drafting and filing, procedural and relief strategy, reconciling authorities and exceptions, and ethical, accountable judgment.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 5 of its 23 references that are in the map. First posted on arXiv 19 Oct 2025.

Builds on

5 of 23 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models20 Aug

Other references · 4

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Modelscited by 70
  3. 2022A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challengescited by 50
  4. 2023A Comprehensive Evaluation of Large Language Models on Legal Judgment Predictioncited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directions
  2. 2026Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task
  3. 2025Structured Definitions and Segmentations for Legal Reasoning in LLMs: A Study on Indian Legal Data
  4. 2025LEXam: Benchmarking Legal Reasoning on 340 Law Exams
  5. 2026Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions
  6. 2026Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases
  7. 2024Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models
  8. 2023Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence