mlsanityData to 14 Oct 2026

IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning

arXiv 2407.05399 ↗First posted 7 Jul 2024cs.CL cs.AI cs.LGcited by 16

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Legal systems worldwide are inundated with exponential growth in cases and documents. There is an imminent need to develop NLP and ML techniques for automatically processing and understanding legal documents to streamline the legal system. However, evaluating and comparing various NLP models designed specifically for the legal domain is challenging. This paper addresses this challenge by proposing IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning. IL-TUR contains monolingual (English, Hindi) and multi-lingual (9 Indian languages) domain-specific tasks that address different aspects of the legal system from the point of view of understanding and reasoning over Indian legal documents. We present baseline models (including LLM-based) for each task, outlining the gap between models and the ground truth. To foster further research in the legal domain, we create a leaderboard (available at: https://exploration-lab.github.io/IL-TUR/) where the research community can upload and compare legal text understanding systems.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 23 of its 71 references that are in the map. First posted on arXiv 7 Jul 2024.

Builds on

23 of 71 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models20 Aug

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2023Mistral 7Bcited by 3,164
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  9. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  10. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  11. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
  12. 2019A Survey on Recent Advances in Named Entity Recognition from Deep Learning modelscited by 117
10 more
  1. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 117
  2. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  3. 2020GLGE: A New General Language Generation Evaluation Benchmarkcited by 47
  4. 2022A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Predictioncited by 44
  5. 2019Identification of Rhetorical Roles of Sentences in Indian Legal Judgmentscited by 39
  6. 2022Pre-trained Language Models for the Legal Domain: A Case Study on Indian Lawcited by 39
  7. 2023ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmarkcited by 33
  8. 2018Natural Environment Benchmarks for Reinforcement Learningcited by 32
  9. 2023MILPaC: A Novel Benchmark for Evaluating Translation of Legal Text to Indian Languagescited by 4
  10. 2020A Divide-and-Conquer Approach to the Summarization of Long Documentscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 6.

16 papers.

Sort:

By area:

  1. 2025ParamBench: A Graduate-Level Benchmark for Evaluating LLM Understanding on Indic Subjectscited by 3
  2. 2025LegalBench.PT: A Benchmark for Portuguese Lawcited by 2
  3. 2025Computational Law: Datasets, Benchmarks, and Ontologiescited by 2
  4. 2024Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Studycited by 1
  5. 2025JBE-QA: Japanese Bar Exam QA Dataset for Assessing Legal Domain Knowledgecited by 1
  6. 2026Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directionscited by 1
  7. 2026GreekBarRetrieval: A Benchmark for Greek Statutory Retrievalcited by 1
  8. 2025textitGrahak-Nyay: Consumer Grievance Redressal through Large Language Modelscited by 0
  9. 2025Nyay-Darpan: Enhancing Decision Making Through Summarization and Case Retrieval for Consumer Law in Indiacited by 0
  10. 2025ReGal: A First Look at PPO-based Legal AI for Judgment Prediction and Summarization in Indiacited by 0
  11. 2025From Scratch to Fine-Tuned: A Comparative Study of Transformer Training Strategies for Legal Machine Translationcited by 0
  12. 2025Domain-Partitioned Hybrid RAG for Legal Reasoning: Toward Modular and Explainable Legal AI for Indiacited by 0
4 more
  1. 2026Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarizationcited by 0
  2. 2026BLAD: A Historically Contextualized, Multilingual Dataset of Bangladeshi Legal Acts (1799 to 2025)cited by 0
  3. 2026Do Small Models Use the Law You Give Them? Context-Injected Fine-Tuning for Legal QA in Bangladeshcited by 0
  4. 2026How Much is a Human Right Worth? ECtHR-NPD: A Benchmark for Predicting Non-Pecuniary Damage Awardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support
  2. 2026Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions
  3. 2025Structured Definitions and Segmentations for Legal Reasoning in LLMs: A Study on Indian Legal Data
  4. 2026AILQA: Evaluating AI-Driven Legal Question Answering Systems for the Indian Legal System
  5. 2026UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoning
  6. 2025Scaling Legal AI: Benchmarking Mamba and Transformers for Statutory Classification and Case Law Retrieval
  7. 2026ILSIC: Corpora for Identifying Indian Legal Statutes from Queries by Laypeople
  8. 2025LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Text