mlsanityData to 14 Oct 2026

Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments

arXiv 2507.04037 ↗First posted 5 Jul 2025cs.AIcited by 13

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

The gap between static benchmarks and the dynamic nature of real-world legal practice poses a key barrier to advancing legal intelligence. To this end, we introduce J1-ENVS, the first interactive and dynamic legal environment tailored for LLM-based agents. Guided by legal experts, it comprises six representative scenarios from Chinese legal practices across three levels of environmental complexity. We further introduce J1-EVAL, a fine-grained evaluation framework, designed to assess both task performance and procedural compliance across varying levels of legal proficiency. Extensive experiments on 17 LLM agents reveal that, while many models demonstrate solid legal knowledge, they struggle with procedural execution in dynamic settings. Even the SOTA model, GPT-4o, falls short of 60% overall performance. These findings highlight persistent challenges in achieving dynamic legal intelligence and offer valuable insights to guide future research.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 32 of its 46 references that are in the map. First posted on arXiv 5 Jul 2025.

Builds on

32 of 46 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025SocioVerse: A World Model for Social Simulation Powered by LLM Agents and A Pool of 10 Million Real-World Users14 Apr
  2. 2025Multi-Agent Simulator Drives Language Models for Legal Intensive Interaction8 Feb
  3. 2024AgentCourt: Simulating Court with Adversarial Evolvable Lawyer Agents15 Aug
  4. 2024Identity-Driven Hierarchical Role-Playing Agents28 Jul
  5. 2023Character-LLM: A Trainable Agent for Role-Playing16 Oct
  6. 2023Generative Agents: Interactive Simulacra of Human Behavior7 Apr

Other references · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  8. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  9. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  10. 2024InternLM2 Technical Reportcited by 397
  11. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  12. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 336
14 more
  1. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  2. 2022Evaluating and Inducing Personality in Pre-trained Language Modelscited by 123
  3. 2023DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Servicescited by 120
  4. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  5. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 117
  6. 2019JEC-QA: A Legal-Domain Question Answering Datasetcited by 116
  7. 2024AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulatorcited by 101
  8. 2024AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarioscited by 40
  9. 2024LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Modelscited by 37
  10. 2022LEVEN: A Large-Scale Chinese Legal Event Detection Datasetcited by 36
  11. 2019Charge-Based Prison Term Prediction with Deep Gating Networkcited by 31
  12. 2024Synergistic Multi-Agent Framework with Trajectory Learning for Knowledge-Intensive Taskscited by 23
  13. 2021Dependency Learning for Legal Judgment Prediction with a Unified Text-to-Text Transformercited by 13
  14. 2019Discrete Argument Representation Learning for Interactive Argument Pair Identificationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 10.

13 papers.

Sort:

By area:

  1. 2026LLM Agents in Law: Taxonomy, Applications, and Challengescited by 5
  2. 2025LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligencecited by 4
  3. 2025CP-Env: Evaluating Large Language Models on Clinical Pathways in a Controllable Hospital Environmentcited by 4
  4. 2026LawThinker: A Deep Research Legal Agent in Dynamic Environmentscited by 4
  5. 2026Unified-MAS: Universally Generating Domain-Specific Nodes for Empowering Automatic Multi-Agent Systemscited by 4
  6. 2026From Knowing to Acting: Benchmarking Self-Awareness Capability of LLM Agentscited by 2
  7. 2025EU-Agent-Bench: Measuring Illegal Behavior of LLM Agents Under EU Lawcited by 1
  8. 2026Beyond Isolated Behaviors: Hierarchical User Modeling for LLM Personalizationcited by 1
  9. 2026LegalWorld: A Life-Cycle Interactive Environment for Legal Agentscited by 1
  10. 2026TriBench-Ko: Evaluating LLM Risks in Judicial Workflowscited by 0
  11. 2026DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultationcited by 0
  12. 2026LexAgentHallu: A Hierarchical Benchmark for Profiling Hallucinations in Legal Agentscited by 0
1 more
  1. 2026ParaAgent: Reinforcing Parallel Acting in Open-World Tool Environmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LegalAgentBench: Evaluating LLM Agents in Legal Domain
  2. 2026LexKairos: Benchmarking Legal Temporal Capabilities in LLMs
  3. 2025Law in Silico: Simulating Legal Society with LLM-Based Agents
  4. 2025MASLegalBench: Benchmarking Multi-Agent Systems in Deductive Legal Reasoning
  5. 2023Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence
  6. 2025Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyond
  7. 2026Can LLMs Time Travel? Enhancing Temporal Consistency in Legal Agentic Search through Reinforcement Learning
  8. 2026BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law