Are LLMs Court-Ready? Evaluating Frontier Models on Indian Legal Reasoning
On the map: LLM reasoning → Reasoning for agents, planning and code
Abstract (arXiv)
Large language models (LLMs) are entering legal workflows, yet we lack a jurisdiction-specific framework to assess their baseline competence therein. We use India's public legal examinations as a transparent proxy. Our multi-year benchmark assembles objective screens from top national and state exams and evaluates open and frontier LLMs under real-world exam conditions. To probe beyond multiple-choice questions, we also include a lawyer-graded, paired-blinded study of long-form answers from the Supreme Court's Advocate-on-Record exam. This is, to our knowledge, the first exam-grounded, India-specific yardstick for LLM court-readiness released with datasets and protocols. Our work shows that while frontier systems consistently clear historical cutoffs and often match or exceed recent top-scorer bands on objective exams, none surpasses the human topper on long-form reasoning. Grader notes converge on three reliability failure modes: procedural or format compliance, authority or citation discipline, and forum-appropriate voice and structure. These findings delineate where LLMs can assist (checks, cross-statute consistency, statute and precedent lookups) and where human leadership remains essential: forum-specific drafting and filing, procedural and relief strategy, reconciling authorities and exceptions, and ethical, accountable judgment.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 5 of its 23 references that are in the map. First posted on arXiv 19 Oct 2025.
Builds on
5 of 23 references.
Closest ancestors · 1
Same direction, newest first
- 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models20 Aug
Other references · 4
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Modelscited by 70
- 2022A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challengescited by 50
- 2023A Comprehensive Evaluation of Large Language Models on Legal Judgment Predictioncited by 28
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directions
- 2026Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task
- 2025Structured Definitions and Segmentations for Legal Reasoning in LLMs: A Study on Indian Legal Data
- 2025LEXam: Benchmarking Legal Reasoning on 340 Law Exams
- 2026Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions
- 2026Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases
- 2024Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models
- 2023Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence