mlsanityData to 14 Oct 2026

DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation

arXiv 2606.13931 ↗First posted 11 Jun 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Lawyer-client consultation is a critical starting point for legal services. Effective legal assistance hinges on eliciting sufficient and truthful information from clients in order to devise strategies that best protect their interests. This task requires Large Language Models (LLMs) not only to perform robust legal reasoning, but also to strategically elicit material facts through multi-turn interactions and effectively guide clients with diverse personalities. Yet existing legal benchmarks overlook this interactive capability. To fill this gap, we introduce DLawBench, a diagnostic benchmark for real-world legal consultation. Drawing on realistic client behavior, we characterize lawyer-client interactions into four types: Cooperative, Dependent, Withdrawn, and Adversarial. Using dialogues grounded in real cases, DLawBench evaluates whether LLMs can effectively conduct legal consultation under realistic conditions. DLawBench comprises 461 cases from Chinese and U.S. law, 5,532 paired fact entries, 3,411 inquiry rubrics, and 3,348 issue-resolution rubrics, and evaluates 26 representative LLMs. Systematic experiments show substantial headroom: the best-performing model, GPT-5.5, achieves only 0.562 on consultation-grounded legal reasoning. More importantly, DLawBench exposes both sycophancy in legal consultation and a paradox: models perform worse when clients need guidance most.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 51 references that are in the map. First posted on arXiv 11 Jun 2026.

Builds on

25 of 51 references.

References · 25

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  4. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  6. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  7. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  8. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  9. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  10. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  11. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  12. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 231
13 more
  1. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 180
  2. 2023InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviewscited by 179
  3. 2024CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluationcited by 143
  4. 2024LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domaincited by 118
  5. 2024LegalAgentBench: Evaluating LLM Agents in Legal Domaincited by 59
  6. 2024BIG5-CHAT: Shaping LLM Personalities Through Training on Human-Grounded Datacited by 50
  7. 2025Multi-Agent Simulator Drives Language Models for Legal Intensive Interactioncited by 22
  8. 2025LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversationcited by 19
  9. 2025Synthetic Socratic Debates: Examining Persona Effects on Moral Decision and Persuasion Dynamicscited by 19
  10. 2025Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environmentscited by 13
  11. 2023Intention and Context Elicitation with Large Language Models in the Legal Aid Intake Processcited by 4
  12. 2026OpenExempt: A Diagnostic Benchmark for Legal Reasoning and a Framework for Creating Custom Benchmarks on Demandcited by 3
  13. 2025LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluationcited by 2

Built on by

Not cited yet.