# IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning

arXiv: https://arxiv.org/abs/2407.05399 · HTML: https://mlsanity.com/paper/2407.05399/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 23 of its 71 references that are in the map.
- First posted on arXiv 7 Jul 2024.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [URLB: Unsupervised Reinforcement Learning Benchmark](https://mlsanity.com/paper/2110.15191/) (2021)
- [A Survey on Recent Advances in Named Entity Recognition from Deep Learning models](https://mlsanity.com/paper/1910.11470/) (2019)
- [LawBench: Benchmarking Legal Knowledge of Large Language Models](https://mlsanity.com/paper/2309.16289/) (2023)
- [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023)
- [GLGE: A New General Language Generation Evaluation Benchmark](https://mlsanity.com/paper/2011.11928/) (2020)
- [A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Prediction](https://mlsanity.com/paper/2206.05224/) (2022)
- [Identification of Rhetorical Roles of Sentences in Indian Legal Judgments](https://mlsanity.com/paper/1911.05405/) (2019)
- [Pre-trained Language Models for the Legal Domain: A Case Study on Indian Law](https://mlsanity.com/paper/2209.06049/) (2022)
- [ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmark](https://mlsanity.com/paper/2304.12202/) (2023)
- [Natural Environment Benchmarks for Reinforcement Learning](https://mlsanity.com/paper/1811.06032/) (2018)
- [MILPaC: A Novel Benchmark for Evaluating Translation of Legal Text to Indian Languages](https://mlsanity.com/paper/2310.09765/) (2023)
- [A Divide-and-Conquer Approach to the Summarization of Long Documents](https://mlsanity.com/paper/2004.06190/) (2020)

## Built on by (most cited first, 16 of 16)

- [ParamBench: A Graduate-Level Benchmark for Evaluating LLM Understanding on Indic Subjects](https://mlsanity.com/paper/2508.16185/) (2025)
- [LegalBench.PT: A Benchmark for Portuguese Law](https://mlsanity.com/paper/2502.16357/) (2025)
- [Computational Law: Datasets, Benchmarks, and Ontologies](https://mlsanity.com/paper/2503.04305/) (2025)
- [Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study](https://mlsanity.com/paper/2412.06272/) (2024)
- [JBE-QA: Japanese Bar Exam QA Dataset for Assessing Legal Domain Knowledge](https://mlsanity.com/paper/2511.22869/) (2025)
- [Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/2601.15267/) (2026)
- [GreekBarRetrieval: A Benchmark for Greek Statutory Retrieval](https://mlsanity.com/paper/2608.18752/) (2026)
- [textitGrahak-Nyay: Consumer Grievance Redressal through Large Language Models](https://mlsanity.com/paper/2507.04854/) (2025)
- [Nyay-Darpan: Enhancing Decision Making Through Summarization and Case Retrieval for Consumer Law in India](https://mlsanity.com/paper/2507.06090/) (2025)
- [ReGal: A First Look at PPO-based Legal AI for Judgment Prediction and Summarization in India](https://mlsanity.com/paper/2512.18014/) (2025)
- [From Scratch to Fine-Tuned: A Comparative Study of Transformer Training Strategies for Legal Machine Translation](https://mlsanity.com/paper/2512.18593/) (2025)
- [Domain-Partitioned Hybrid RAG for Legal Reasoning: Toward Modular and Explainable Legal AI for India](https://mlsanity.com/paper/2602.23371/) (2025)
- [Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization](https://mlsanity.com/paper/2605.17379/) (2026)
- [BLAD: A Historically Contextualized, Multilingual Dataset of Bangladeshi Legal Acts (1799 to 2025)](https://mlsanity.com/paper/2607.17111/) (2026)
- [Do Small Models Use the Law You Give Them? Context-Injected Fine-Tuning for Legal QA in Bangladesh](https://mlsanity.com/paper/2607.23446/) (2026)
- [How Much is a Human Right Worth? ECtHR-NPD: A Benchmark for Predicting Non-Pecuniary Damage Awards](https://mlsanity.com/paper/2609.18908/) (2026)

## Abstract (arXiv)

Legal systems worldwide are inundated with exponential growth in cases and documents. There is an imminent need to develop NLP and ML techniques for automatically processing and understanding legal documents to streamline the legal system. However, evaluating and comparing various NLP models designed specifically for the legal domain is challenging. This paper addresses this challenge by proposing IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning. IL-TUR contains monolingual (English, Hindi) and multi-lingual (9 Indian languages) domain-specific tasks that address different aspects of the legal system from the point of view of understanding and reasoning over Indian legal documents. We present baseline models (including LLM-based) for each task, outlining the gap between models and the ground truth. To foster further research in the legal domain, we create a leaderboard (available at: https://exploration-lab.github.io/IL-TUR/) where the research community can upload and compare legal text understanding systems.