mlsanityData to 7 Oct 2026

R-Judge: Benchmarking Safety Risk Awareness for LLM Agents

arXiv 2401.10019 ↗First posted 18 Jan 2024cs.CL cs.AIcited by 268

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large language models (LLMs) have exhibited great potential in autonomously completing tasks across real-world applications. Despite this, these LLM agents introduce unexpected safety risks when operating in interactive environments. Instead of centering on the harmlessness of LLM-generated content in most prior studies, this work addresses the imperative need for benchmarking the behavioral safety of LLM agents within diverse environments. We introduce R-Judge, a benchmark crafted to evaluate the proficiency of LLMs in judging and identifying safety risks given agent interaction records. R-Judge comprises 569 records of multi-turn agent interaction, encompassing 27 key risk scenarios among 5 application categories and 10 risk types. It is of high-quality curation with annotated safety labels and risk descriptions. Evaluation of 11 LLMs on R-Judge shows considerable room for enhancing the risk awareness of LLMs: The best-performing model, GPT-4o, achieves 74.42% while no other models significantly exceed the random. Moreover, we reveal that risk awareness in open agent scenarios is a multi-dimensional capability involving knowledge and reasoning, thus challenging for LLMs. With further experiments, we find that fine-tuning on safety judgment significantly improve model performance while straightforward prompting mechanisms fail. R-Judge is publicly available at https://github.com/Lordog/R-Judge.

Key facts. Cited by 268 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 63 of its 91 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

63 of 91 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Evil Geniuses: Delving into the Safety of LLM-based Agents20 Nov
  2. 2023Testing Language Model Agents Safely in the Wild17 Nov
  3. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 60

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
38 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  5. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  11. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  12. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  13. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  14. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  15. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  16. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  17. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  18. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  19. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  20. 2023Cognitive Architectures for Language Agentscited by 422
  21. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  22. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  23. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  24. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  25. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  26. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  27. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  28. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  29. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  30. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  31. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  32. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 172
  33. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 166
  34. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 136
  35. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 130
  36. 2023Agents: An Open-source Framework for Autonomous Language Agentscited by 116
  37. 2020MuTual: A Dataset for Multi-Turn Dialogue Reasoningcited by 114
  38. 2023Can Large Language Models Really Improve by Self-critiquing Their Own Plans?cited by 107

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 44; 2025: 77; 2026: 147.

268 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  3. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  4. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  5. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  6. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  7. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 126
  8. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  9. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  10. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoningcited by 113
  11. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  12. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
38 more
  1. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 104
  2. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 102
  3. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 98
  4. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  5. 2024RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graphcited by 88
  6. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agentscited by 87
  7. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  8. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 86
  9. 2025A Survey of LLM-based Agents in Medicine: How far are we from Baymax?cited by 84
  10. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 82
  11. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 77
  12. 2024Safeguarding Large Language Models: A Surveycited by 73
  13. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 71
  14. 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automationcited by 70
  15. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  16. 2024CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environmentscited by 69
  17. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agentscited by 67
  18. 2024SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Typescited by 58
  19. 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agentscited by 57
  20. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 55
  21. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  22. 2024MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Controlcited by 49
  23. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  24. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  25. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 46
  26. 2024Measuring Bargaining Abilities of LLMs: A Benchmark and A Buyer-Enhancement Methodcited by 44
  27. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
  28. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 42
  29. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 42
  30. 2025Magentic-UI: Towards Human-in-the-loop Agentic Systemscited by 41
  31. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agentscited by 34
  32. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 30
  33. 2026ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosiscited by 30
  34. 2024S-Eval: Towards Automated and Comprehensive Safety Evaluation for Large Language Modelscited by 28
  35. 2025LLM Agents Should Employ Security Principlescited by 28
  36. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  37. 2024Can Watermarks Survive Translation? On the Cross-lingual Consistency of Text Watermark for Large Language Modelscited by 27
  38. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26

50 most cited of 268 All 268 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient LLM Safety Evaluation through Multi-Agent Debate
  2. 2025Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon Simulation
  3. 2025SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarking
  4. 2025ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMs
  5. 2026The Cold-Start Safety Gap in LLM Agents
  6. 2025A Framework for Benchmarking and Aligning Task-Planning Safety in LLM-Based Embodied Agents
  7. 2025RADAR: A Risk-Aware Dynamic Multi-Agent Framework for LLM Safety Evaluation via Role-Specialized Collaboration
  8. 2024Large Language Model Safety: A Holistic Survey