# Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments

arXiv: https://arxiv.org/abs/2507.04037 · HTML: https://mlsanity.com/paper/2507.04037/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 32 of its 46 references that are in the map.
- First posted on arXiv 5 Jul 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [InternLM2 Technical Report](https://mlsanity.com/paper/2403.17297/) (2024)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [Character-LLM: A Trainable Agent for Role-Playing](https://mlsanity.com/paper/2310.10158/) (2023)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [Evaluating and Inducing Personality in Pre-trained Language Models](https://mlsanity.com/paper/2206.07550/) (2022)
- [DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services](https://mlsanity.com/paper/2309.11325/) (2023)
- [AgentGym: Evolving Large Language Model-based Agents across Diverse Environments](https://mlsanity.com/paper/2406.04151/) (2024)
- [LawBench: Benchmarking Legal Knowledge of Large Language Models](https://mlsanity.com/paper/2309.16289/) (2023)
- [JEC-QA: A Legal-Domain Question Answering Dataset](https://mlsanity.com/paper/1911.12011/) (2019)
- [AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator](https://mlsanity.com/paper/2402.09742/) (2024)
- [SocioVerse: A World Model for Social Simulation Powered by LLM Agents and A Pool of 10 Million Real-World Users](https://mlsanity.com/paper/2504.10157/) (2025)
- [AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios](https://mlsanity.com/paper/2410.19346/) (2024)
- [AgentCourt: Simulating Court with Adversarial Evolvable Lawyer Agents](https://mlsanity.com/paper/2408.08089/) (2024)
- [LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models](https://mlsanity.com/paper/2409.20288/) (2024)
- [LEVEN: A Large-Scale Chinese Legal Event Detection Dataset](https://mlsanity.com/paper/2203.08556/) (2022)
- [Charge-Based Prison Term Prediction with Deep Gating Network](https://mlsanity.com/paper/1908.11521/) (2019)
- [Synergistic Multi-Agent Framework with Trajectory Learning for Knowledge-Intensive Tasks](https://mlsanity.com/paper/2407.09893/) (2024)
- [Multi-Agent Simulator Drives Language Models for Legal Intensive Interaction](https://mlsanity.com/paper/2502.06882/) (2025)
- [Identity-Driven Hierarchical Role-Playing Agents](https://mlsanity.com/paper/2407.19412/) (2024)

## Built on by (most cited first, 13 of 13)

- [LLM Agents in Law: Taxonomy, Applications, and Challenges](https://mlsanity.com/paper/2601.06216/) (2026)
- [LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligence](https://mlsanity.com/paper/2512.04578/) (2025)
- [CP-Env: Evaluating Large Language Models on Clinical Pathways in a Controllable Hospital Environment](https://mlsanity.com/paper/2512.10206/) (2025)
- [LawThinker: A Deep Research Legal Agent in Dynamic Environments](https://mlsanity.com/paper/2602.12056/) (2026)
- [Unified-MAS: Universally Generating Domain-Specific Nodes for Empowering Automatic Multi-Agent Systems](https://mlsanity.com/paper/2603.21475/) (2026)
- [From Knowing to Acting: Benchmarking Self-Awareness Capability of LLM Agents](https://mlsanity.com/paper/2606.20661/) (2026)
- [EU-Agent-Bench: Measuring Illegal Behavior of LLM Agents Under EU Law](https://mlsanity.com/paper/2510.21524/) (2025)
- [Beyond Isolated Behaviors: Hierarchical User Modeling for LLM Personalization](https://mlsanity.com/paper/2606.02300/) (2026)
- [LegalWorld: A Life-Cycle Interactive Environment for Legal Agents](https://mlsanity.com/paper/2606.18728/) (2026)
- [TriBench-Ko: Evaluating LLM Risks in Judicial Workflows](https://mlsanity.com/paper/2605.03792/) (2026)
- [DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation](https://mlsanity.com/paper/2606.13931/) (2026)
- [LexAgentHallu: A Hierarchical Benchmark for Profiling Hallucinations in Legal Agents](https://mlsanity.com/paper/2609.09754/) (2026)
- [ParaAgent: Reinforcing Parallel Acting in Open-World Tool Environments](https://mlsanity.com/paper/2609.33618/) (2026)

## Abstract (arXiv)

The gap between static benchmarks and the dynamic nature of real-world legal practice poses a key barrier to advancing legal intelligence. To this end, we introduce J1-ENVS, the first interactive and dynamic legal environment tailored for LLM-based agents. Guided by legal experts, it comprises six representative scenarios from Chinese legal practices across three levels of environmental complexity. We further introduce J1-EVAL, a fine-grained evaluation framework, designed to assess both task performance and procedural compliance across varying levels of legal proficiency. Extensive experiments on 17 LLM agents reveal that, while many models demonstrate solid legal knowledge, they struggle with procedural execution in dynamic settings. Even the SOTA model, GPT-4o, falls short of 60% overall performance. These findings highlight persistent challenges in achieving dynamic legal intelligence and offer valuable insights to guide future research.