mlsanityData to 7 Oct 2026

LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversation

arXiv 2502.20640 ↗First posted 28 Feb 2025cs.CL cs.IRcited by 19

On the map: Retrieval-augmented generation → Legal and financial RAG

Abstract (arXiv)

Retrieval-augmented generation (RAG) has proven highly effective in improving large language models (LLMs) across various domains. However, there is no benchmark specifically designed to assess the effectiveness of RAG in the legal domain, which restricts progress in this area. To fill this gap, we propose LexRAG, the first benchmark to evaluate RAG systems for multi-turn legal consultations. LexRAG consists of 1,013 multi-turn dialogue samples and 17,228 candidate legal articles. Each sample is annotated by legal experts and consists of five rounds of progressive questioning. LexRAG includes two key tasks: (1) Conversational knowledge retrieval, requiring accurate retrieval of relevant legal articles based on multi-turn context. (2) Response generation, focusing on producing legally sound answers. To ensure reliable reproducibility, we develop LexiT, a legal RAG toolkit that provides a comprehensive implementation of RAG system components tailored for the legal domain. Additionally, we introduce an LLM-as-a-judge evaluation pipeline to enable detailed and effective assessment. Through experimental analysis of various LLMs and retrieval methods, we reveal the key limitations of existing RAG systems in handling legal consultation conversations. LexRAG establishes a new benchmark for the practical application of RAG systems in the legal domain, with its code and data available at https://github.com/CSHaitao/LexRAG.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Legal and financial RAG, one of 15 directions in Retrieval-augmented generation. Builds on 31 of its 43 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

31 of 43 references.

References · 31

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2017Billion-scale similarity search with GPUscited by 2,477
  6. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  7. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  8. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  9. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  10. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  11. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  12. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
19 more
  1. 2018CAIL2018: A Large-Scale Legal Dataset for Judgment Predictioncited by 168
  2. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 165
  3. 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
  4. 2019Challenges in Building Intelligent Open-domain Dialog Systemscited by 147
  5. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  6. 2023Large Language Models in Law: A Surveycited by 128
  7. 2023DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Servicescited by 117
  8. 2023SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrievalcited by 61
  9. 2021Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representationscited by 60
  10. 2020KdConv: A Chinese Multi-domain Dialogue Dataset Towards Multi-turn Knowledge-driven Conversationcited by 59
  11. 2023T2Ranking: A large-scale Chinese Benchmark for Passage Rankingcited by 59
  12. 2024LegalAgentBench: Evaluating LLM Agents in Legal Domaincited by 56
  13. 2023LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Datasetcited by 37
  14. 2024CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judgescited by 32
  15. 2024PRE: A Peer Review Based Large Language Model Evaluatorcited by 28
  16. 2024BLADE: Enhancing Black-box Large Language Models with Small Domain-Specific Modelscited by 15
  17. 2024A User-Centric Multi-Intent Benchmark for Evaluating Large Language Modelscited by 15
  18. 2024DELTA: Pre-train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignmentcited by 9
  19. 2024Evaluation Ethics of LLMs in Legal Domaincited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 12.

19 papers.

Sort:

By area:

  1. 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practicecited by 13
  2. 2025Ranking Free RAG: Replacing Re-ranking with Selection in RAG for Sensitive Domainscited by 5
  3. 2025JurisTCU: A Brazilian Portuguese Information Retrieval Dataset with Query Relevance Judgmentscited by 2
  4. 2025Scene-Aware Vectorized Memory Multi-Agent Framework with Cross-Modal Differentiated Quantization VLMs for Visually Impaired Assistancecited by 1
  5. 2025From Answers to Guidance: A Proactive Dialogue System for Legal Documentscited by 1
  6. 2026LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Taskscited by 1
  7. 2025ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in Game RAG Benchmarkscited by 0
  8. 2025R2R: A Route-to-Rerank Post-Training Framework for Multi-Domain Decoder-Only Rerankerscited by 0
  9. 2025Domain-Partitioned Hybrid RAG for Legal Reasoning: Toward Modular and Explainable Legal AI for Indiacited by 0
  10. 2026Less Data Approximates More: Earning Faithful Confidence in High-Stakes Domainscited by 0
  11. 2026Navigating Global AI Regulation: A Multi-Jurisdictional Retrieval-Augmented Generation Systemcited by 0
  12. 2026MAP-Law: Coverage-Driven Retrieval Control for Multi-Turn Legal Consultationcited by 0
7 more
  1. 2026Civil Court Simulation with Large Language Modelscited by 0
  2. 2026DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultationcited by 0
  3. 2026Evaluating RAG for French immigration law: a benchmark and baseline studycited by 0
  4. 2026Efficient LLM Distillation for Bangladesh Legal Context: A Smartphone-Compatible Retrieval-Augmented Generation Modelcited by 0
  5. 2026LEGO: Synergizing Expert GraphRAG and Expert Chain-of-Thought for Legal Reasoningcited by 0
  6. 2026EMIR2: Evolution-Aware Memory with Intent-Guided Multi-Round Retrievalcited by 0
  7. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Fine-grained Claim-level RAG Benchmark for Law
  2. 2024LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain
  3. 2026Legal-DC: Benchmarking Retrieval-Augmented Generation for Legal Documents
  4. 2025The 3rd Place Solution of CCIR CUP 2025: A Framework for Retrieval-Augmented Generation in Multi-Turn Legal Conversation
  5. 2025All for law and law for all: Adaptive RAG Pipeline for Legal Research
  6. 2025A Reasoning-Focused Legal Retrieval Benchmark
  7. 2025LRAGE: Legal Retrieval Augmented Generation Evaluation Tool
  8. 2024CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering