# LRAGE: Legal Retrieval Augmented Generation Evaluation Tool

arXiv: https://arxiv.org/abs/2504.01840 · HTML: https://mlsanity.com/paper/2504.01840/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Legal and financial RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Placed in

- Retrieval-augmented generation / [Legal and financial RAG](https://mlsanity.com/rag/legal-and-financial-rag/)

## Builds on (most cited first, 24 of 24)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [Multilingual E5 Text Embeddings: A Technical Report](https://mlsanity.com/paper/2402.05672/) (2024)
- [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models](https://mlsanity.com/paper/2401.00396/) (2023)
- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024)
- [CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction](https://mlsanity.com/paper/1807.02478/) (2018)
- [LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain](https://mlsanity.com/paper/2408.10343/) (2024)
- [Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset](https://mlsanity.com/paper/2207.00220/) (2022)
- [The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models](https://mlsanity.com/paper/2406.05761/) (2024)
- [SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval](https://mlsanity.com/paper/2304.11370/) (2023)
- [LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development](https://mlsanity.com/paper/2305.07507/) (2023)
- [A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Prediction](https://mlsanity.com/paper/2206.05224/) (2022)
- [LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Dataset](https://mlsanity.com/paper/2310.17609/) (2023)
- [BERGEN: A Benchmarking Library for Retrieval-Augmented Generation](https://mlsanity.com/paper/2407.01102/) (2024)
- [EXAONE 3.0 7.8B Instruction Tuned Language Model](https://mlsanity.com/paper/2408.03541/) (2024)
- [CLERC: A Dataset for Legal Case Retrieval and Retrieval-Augmented Analysis Generation](https://mlsanity.com/paper/2406.17186/) (2024)
- [Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer?](https://mlsanity.com/paper/2310.14880/) (2023)
- [Enhancing Legal Case Retrieval via Scaling High-quality Synthetic Query-Candidate Pairs](https://mlsanity.com/paper/2410.06581/) (2024)
- [rerankers: A Lightweight Python Library to Unify Ranking Methods](https://mlsanity.com/paper/2408.17344/) (2024)
- [Taxation Perspectives from Large Language Models: A Case Study on Additional Tax Penalties](https://mlsanity.com/paper/2503.03444/) (2025)

## Built on by (most cited first, 5 of 5)

- [Fine-grained Claim-level RAG Benchmark for Law](https://mlsanity.com/paper/2605.21071/) (2026)
- [Can LLMs Time Travel? Enhancing Temporal Consistency in Legal Agentic Search through Reinforcement Learning](https://mlsanity.com/paper/2605.25920/) (2026)
- [Na Pr'atica, qual IA Entende o Direito? Um Estudo Experimental com IAs Generalistas e uma IA Jur'idica](https://mlsanity.com/paper/2510.18108/) (2025)
- [Chinese Labor Law Large Language Model Benchmark](https://mlsanity.com/paper/2601.09972/) (2026)
- [Can Small Models Reason About Legal Documents? A Comparative Study](https://mlsanity.com/paper/2603.25944/) (2026)

## Abstract (arXiv)

Recently, building retrieval-augmented generation (RAG) systems to enhance the capability of large language models (LLMs) has become a common practice. Especially in the legal domain, previous judicial decisions play a significant role under the doctrine of stare decisis which emphasizes the importance of making decisions based on (retrieved) prior documents. However, the overall performance of RAG system depends on many components: (1) retrieval corpora, (2) retrieval algorithms, (3) rerankers, (4) LLM backbones, and (5) evaluation metrics. Here we propose LRAGE, an open-source tool for holistic evaluation of RAG systems focusing on the legal domain. LRAGE provides GUI and CLI interfaces to facilitate seamless experiments and investigate how changes in the aforementioned five components affect the overall accuracy. We validated LRAGE using multilingual legal benches including Korean (KBL), English (LegalBench), and Chinese (LawBench) by demonstrating how the overall accuracy changes when varying the five components mentioned above. The source code is available at https://github.com/hoorangyee/LRAGE.