mlsanityData to 14 Oct 2026

LegalAgentBench: Evaluating LLM Agents in Legal Domain

arXiv 2412.17259 ↗First posted 23 Dec 2024cs.CL cs.IRcited by 59

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

With the increasing intelligence and autonomy of LLM agents, their potential applications in the legal domain are becoming increasingly apparent. However, existing general-domain benchmarks cannot fully capture the complexity and subtle nuances of real-world judicial cognition and decision-making. Therefore, we propose LegalAgentBench, a comprehensive benchmark specifically designed to evaluate LLM Agents in the Chinese legal domain. LegalAgentBench includes 17 corpora from real-world legal scenarios and provides 37 tools for interacting with external knowledge. We designed a scalable task construction framework and carefully annotated 300 tasks. These tasks span various types, including multi-hop reasoning and writing, and range across different difficulty levels, effectively reflecting the complexity of real-world legal scenarios. Moreover, beyond evaluating final success, LegalAgentBench incorporates keyword analysis during intermediate processes to calculate progress rates, enabling more fine-grained evaluation. We evaluated eight popular LLMs, highlighting the strengths, limitations, and potential areas for improvement of existing models and methods. LegalAgentBench sets a new benchmark for the practical application of LLMs in the legal domain, with its code and data available at urlhttps://github.com/CSHaitao/LegalAgentBench.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 21 of its 32 references that are in the map. First posted on arXiv 23 Dec 2024.

Builds on

21 of 32 references.

References · 21

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2023Qwen Technical Reportcited by 3,344
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  8. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  9. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  10. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  11. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 274
  12. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 217
9 more
  1. 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 93
  2. 2023SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrievalcited by 62
  3. 2023T2Ranking: A large-scale Chinese Benchmark for Passage Rankingcited by 60
  4. 2023LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Datasetcited by 38
  5. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarioscited by 35
  6. 2024CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judgescited by 34
  7. 2024PRE: A Peer Review Based Large Language Model Evaluatorcited by 28
  8. 2024BLADE: Enhancing Black-box Large Language Models with Small Domain-Specific Modelscited by 15
  9. 2024DELTA: Pre-train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignmentcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 31; 2026: 28.

59 papers.

Sort:

By area:

  1. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  2. 2025SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agentscited by 69
  3. 2025LLM Agents for Education: Advances and Applicationscited by 59
  4. 2025LLM Agents Making Agent Toolscited by 48
  5. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 34
  6. 2025A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrievalcited by 33
  7. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  8. 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systemscited by 26
  9. 2025LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversationcited by 19
  10. 2026DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivitycited by 14
  11. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injectioncited by 14
  12. 2025Enhancing LLM-Based Agents via Global Planning and Hierarchical Executioncited by 13
38 more
  1. 2026DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Usecited by 13
  2. 2025FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agentscited by 12
  3. 2025CaseGen: A Benchmark for Multi-Stage Legal Case Documents Generationcited by 8
  4. 2025KoBLEX: Open Legal Question Answering with Multi-hop Reasoningcited by 7
  5. 2025AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliancecited by 6
  6. 2025Can LLMs Refuse Questions They Do Not Know? Measuring Knowledge-Aware Refusal in Factual Taskscited by 6
  7. 2025ModelingAgent: Bridging LLMs and Mathematical Modeling for Real-World Challengescited by 5
  8. 2026LLM Agents in Law: Taxonomy, Applications, and Challengescited by 5
  9. 2026ADAM: A Systematic Data Extraction Attack on Agent Memory via Adaptive Queryingcited by 5
  10. 2026FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Qualitycited by 5
  11. 2025LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligencecited by 4
  12. 2026LawThinker: A Deep Research Legal Agent in Dynamic Environmentscited by 4
  13. 2025DICE: A Framework for Dimensional and Contextual Evaluation of Language Modelscited by 3
  14. 2025AppealCase: A Dataset and Benchmark for Civil Case Appeal Scenarioscited by 3
  15. 2025Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHubcited by 3
  16. 2025MASLegalBench: Benchmarking Multi-Agent Systems in Deductive Legal Reasoningcited by 3
  17. 2026ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Modelscited by 3
  18. 2025ChemAU: Harness the Reasoning of LLMs in Chemical Research with Adaptive Uncertainty Estimationcited by 2
  19. 2025TransLaw: A Large-Scale Dataset and Multi-Agent Benchmark Simulating Professional Translation of Hong Kong Case Lawcited by 2
  20. 2025Mina: A Multilingual LLM-Powered Legal Assistant Agent for Bangladesh for Empowering Access to Justicecited by 2
  21. 2026Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?cited by 2
  22. 2026LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Taskscited by 2
  23. 2025Auto-SLURP: A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistantcited by 1
  24. 2025What Would an LLM Do? Evaluating Large Language Models for Policymaking to Alleviate Homelessnesscited by 1
  25. 2026AppellateGen: A Benchmark for Appellate Legal Judgment Generationcited by 1
  26. 2026Parthenon Law: A Self-Evolving Legal-Agent Frameworkcited by 1
  27. 2026Metacognition in LLMs: Foundations, Progress, and Opportunitiescited by 1
  28. 2026WANDR: A Benchmark for Wide and Deep Researchcited by 1
  29. 2026Legal Research Bench: Measuring End-to-End Reliability in Long-Horizon Legal Research Agentscited by 1
  30. 2025Collaborative Editable Modelcited by 0
  31. 2025Using Large Language Models for Legal Decision-Making in Austrian Value-Added Tax Law: A Comparative Studycited by 0
  32. 2025LaQual: An Automated Framework for LLM App Quality Evaluationcited by 0
  33. 2025PL-CA: A Parametric Legal Case Augmentation Frameworkcited by 0
  34. 2025Evaluating Hydro-Science and Engineering Knowledge of Large Language Modelscited by 0
  35. 2026A Cloud-based Multi-Agentic Workflow for Sciencecited by 0
  36. 2026SCORE: Specificity, Context Utilization, Robustness, and Relevance for Reference-Free LLM Evaluationcited by 0
  37. 2026LAMUS: A Large-Scale Corpus for Legal Argument Mining from U.S. Caselaw using LLMscited by 0
  38. 2026RPS: Information Elicitation with Reinforcement Prompt Selectioncited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments
  2. 2023LawBench: Benchmarking Legal Knowledge of Large Language Models
  3. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
  4. 2026BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
  5. 2026Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions
  6. 2026LexKairos: Benchmarking Legal Temporal Capabilities in LLMs
  7. 2025Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyond
  8. 2025LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation