mlsanityData to 7 Oct 2026

TrustAgent: Towards Safe and Trustworthy LLM-based Agents

arXiv 2402.01586 ↗First posted 2 Feb 2024cs.CL cs.AI cs.LGcited by 62

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

The rise of LLM-based agents shows great potential to revolutionize task planning, capturing significant attention. Given that these agents will be integrated into high-stake domains, ensuring their reliability and safety is crucial. This paper presents an Agent-Constitution-based agent framework, TrustAgent, with a particular focus on improving the LLM-based agent safety. The proposed framework ensures strict adherence to the Agent Constitution through three strategic components: pre-planning strategy which injects safety knowledge to the model before plan generation, in-planning strategy which enhances safety during plan generation, and post-planning strategy which ensures safety by post-planning inspection. Our experimental results demonstrate that the proposed framework can effectively enhance an LLM agent's safety across multiple domains by identifying and mitigating potential dangers during the planning. Further analysis reveals that the framework not only improves safety but also enhances the helpfulness of the agent. Additionally, we highlight the importance of the LLM reasoning ability in adhering to the Constitution. This paper sheds light on how to ensure the safe integration of LLM-based agents into human-centric environments. Data and code are available at https://github.com/agiresearch/TrustAgent.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 35 of its 49 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

35 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Evil Geniuses: Delving into the Safety of LLM-based Agents20 Nov
  2. 2023Testing Language Model Agents Safely in the Wild17 Nov
  3. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2024Mixtral of Expertscited by 1,785
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  9. 2023BloombergGPT: A Large Language Model for Financecited by 923
  10. 2020Aligning AI With Shared Human Valuescited by 706
  11. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  12. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
20 more
  1. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  2. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  3. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  4. 2023Explainability for Large Language Models: A Surveycited by 331
  5. 2023Faithful Chain-of-Thought Reasoningcited by 285
  6. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  7. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  8. 2023Large Language Models in Finance: A Surveycited by 223
  9. 2023OpenAGI: When LLM Meets Domain Expertscited by 196
  10. 2023FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Designcited by 163
  11. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  12. 2023War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Warscited by 143
  13. 2023How to Index Item IDs for Recommendation Foundation Modelscited by 102
  14. 2016Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervisioncited by 101
  15. 2023Controlled Text Generation with Natural Language Instructionscited by 87
  16. 2023Humanoid Agents: Platform for Simulating Human-like Generative Agentscited by 73
  17. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  18. 2022ArcaneQA: Dynamic Program Induction and Contextualized Encoding for Knowledge Base Question Answeringcited by 48
  19. 2023UP5: Unbiased Foundation Model for Fairness-aware Recommendationcited by 38
  20. 2023Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanismcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 32; 2026: 23.

62 papers.

Sort:

By area:

  1. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  2. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 104
  3. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  4. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 82
  5. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  6. 2025G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systemscited by 56
  7. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 55
  8. 2024MoralBench: Moral Evaluation of LLMscited by 50
  9. 2024Evaluating Cultural and Social Awareness of LLM Web Agentscited by 43
  10. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 42
  11. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
  12. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
38 more
  1. 2026Towards Verifiably Safe Tool Use for LLM Agentscited by 34
  2. 2025Towards Trustworthy GUI Agents: A Surveycited by 33
  3. 2025MCIP: Protecting MCP Safety via Model Contextual Integrity Protocolcited by 33
  4. 2025LLM Agents Should Employ Security Principlescited by 28
  5. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  6. 2025Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systemscited by 27
  7. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 26
  8. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 26
  9. 2024Multi-Agent Large Language Models for Conversational Task-Solvingcited by 24
  10. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  11. 2025Towards Enforcing Company Policy Adherence in Agentic Workflowscited by 17
  12. 2026Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphscited by 17
  13. 2026The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agentscited by 14
  14. 2025Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspectivecited by 12
  15. 2025Policy-as-Prompt: Turning AI Governance Rules into Guardrails for AI Agentscited by 12
  16. 2024Private Yet Social: How LLM Chatbots Support and Challenge Eating Disorder Recoverycited by 8
  17. 2025Web Fraud Attacks Against LLM-Driven Multi-Agent Systemscited by 8
  18. 2026Defense Against Indirect Prompt Injection via Tool Result Parsingcited by 8
  19. 2026AIR: Improving Agent Safety through Incident Responsecited by 8
  20. 2025TAI3: Testing Agent Integrity in Interpreting User Intentcited by 7
  21. 2025SentinelNet: Safeguarding Multi-Agent Collaboration Through Credit-Based Dynamic Threat Detectioncited by 7
  22. 2025SafeAgent: Safeguarding LLM Agents via an Automated Risk Simulatorcited by 6
  23. 2026Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studiescited by 6
  24. 2025Achilles Heel of Distributed Multi-Agent Systemscited by 5
  25. 2025ALRPHFS: Adversarially Learned Risk Patterns with Hierarchical Fast & Slow Reasoning for Robust Agent Defensecited by 5
  26. 2025LLMPC: Large Language Model Predictive Controlcited by 4
  27. 2025A Practical Approach for Building Production-Grade Conversational Agents with Workflow Graphscited by 4
  28. 2025LumiMAS: A Comprehensive Framework for Real-Time Monitoring and Enhanced Observability in Multi-Agent Systemscited by 4
  29. 2026SpecOps: A Fully Automated AI Agent Testing Framework in Real-World GUI Environmentscited by 4
  30. 2026The Hallucination Snowball: Modeling Error Propagation as State Transitions in Multi-Agent LLM Pipelinescited by 4
  31. 2025Extending the OWASP Multi-Agentic System Threat Modeling Guide: Insights from Multi-Agent Security Researchcited by 3
  32. 2025Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attackcited by 3
  33. 2025Ev-Trust: An Evolutionarily Stable Trust Mechanism for Decentralized LLM-Based Multi-Agent Service Economiescited by 3
  34. 2026Automated structural testing of LLM-based agents: methods, framework, and case studiescited by 3
  35. 2026TopoPilot: Reliable Conversational Workflow Automation for Topological Data Analysis and Visualizationcited by 3
  36. 2025Phonetic Perturbations Reveal Tokenizer-Rooted Safety Gaps in LLMscited by 2
  37. 2026Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAGcited by 2
  38. 2026Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safetycited by 2

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Toward Safe LLM Agents: A Survey of Specification, Verification, and Enforcement
  2. 2025Evaluating LLM Agent Adherence to Hierarchical Safety Principles: A Lightweight Benchmark for Probing Foundational Controllability Components
  3. 2026Can Agents Trust Their Skills? Uncovering Unsafe Chains of Trust in Skill-Based LLM Agents
  4. 2025SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agents
  5. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning
  6. 2024Safeguarding AI Agents: Developing and Analyzing Safety Architectures
  7. 2023Enhancing Trust in LLM-Based AI Automation Agents: New Considerations and Future Challenges
  8. 2025A Framework for Benchmarking and Aligning Task-Planning Safety in LLM-Based Embodied Agents