Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correction
On the map: LLM reasoning → Reasoning for agents, planning and code
also in 2 more
Abstract (arXiv)
LLM-based agents solve complex tasks through iterative reasoning, tool use, and environment interaction, where each intermediate thought directly shapes subsequent actions. Small deviations in these thoughts can therefore propagate into unsafe behaviors, yet existing guardrails typically operate only on final outputs or require intrusive model modifications. We introduce Thought-Aligner, a lightweight plug-in safety model that performs causal correction on unsafe thoughts before action execution, without altering the underlying agent. The corrected thoughts are fed back into the agent, steering its decision process and tool use toward safer trajectories. Because it operates solely at the thought level, Thought-Aligner is model-agnostic and can be integrated into diverse agent frameworks. We train Thought-Aligner via two-stage contrastive learning on paired safe and unsafe thoughts generated across ten risk scenarios. Experiments on diverse agent-safety benchmarks and six LLMs show that Thought-Aligner increases behavioral safety from about 50% without protection to around 90% on average, exceeding state-of-the-art guardrails by roughly 23%, while also improving helpfulness by about 5%. The method incurs low per-step latency and minimal overhead, enabling scalable and practical deployment. We publicly release Thought-Aligner-7B at https://huggingface.co/WhitzardAgent/Thought-Aligner-7B.
Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 59 of its 71 references that are in the map. First posted on arXiv 16 May 2025.
Builds on
59 of 71 references.
Closest ancestors · 2
Same direction, newest first
- 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning26 Mar
- 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning13 Jun
Other references · 57
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
- 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
38 more
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
- 2024Executable Code Actions Elicit Better LLM Agentscited by 564
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
- 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
- 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 538
- 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
- 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 429
- 2023Language Models can Solve Computer Taskscited by 372
- 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 369
- 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
- 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
- 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 306
- 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
- 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
- 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
- 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
- 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 171
- 2023A Language Agent for Autonomous Drivingcited by 164
- 2023FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Designcited by 164
- 2024Certifiably Robust RAG against Retrieval Corruptioncited by 151
- 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
- 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 150
- 2024Autonomous Evaluation and Refinement of Digital Agentscited by 137
- 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 133
- 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 131
- 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 120
- 2024Attacking Vision-Language Computer Agents via Pop-upscited by 116
- 2024Aligner: Efficient Alignment by Learning to Correctcited by 98
- 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 96
- 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 92
- 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agentscited by 90
- 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 82
- 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 81
- 2024EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Recordscited by 78
- 2024Imprompter: Tricking LLM Agents into Improper Tool Usecited by 68
- 2024AdvAgent: Controllable Blackbox Red-teaming on Web Agentscited by 59
- 2023Testing Language Model Agents Safely in the Wildcited by 46
50 most cited of 57 All 57 (JSON)
Built on by
Citing papers per year: 2025: 6; 2026: 8.
14 papers.
Sort:
By area:
- 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 57
- 2025Agentic Web: Weaving the Next Web with AI Agentscited by 45
- 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
- 2026SafeHarness: Lifecycle-Integrated Security Architecture for LLM-based Agent Deploymentcited by 15
- 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 14
- 2025Agent Safety Alignment via Reinforcement Learningcited by 11
- 2025HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimizationcited by 10
- 2026Learning When to Act or Refuse: Guarding Agentic Reasoning Models for Safe Multi-Step Tool Usecited by 8
- 2026MirrorGuard: Toward Secure Computer-Use Agents via Simulation-to-Real Reasoning Correctioncited by 5
- 2025When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?cited by 3
- 2026AgentGuard: An Attribute-Based Access Control Framework for Tool-Use LLM-Based Agentcited by 3
- 2026Governance by Construction for Generalist Agentscited by 1
- 2026AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents27 Sep 2026
- 2026MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning19 Sep 2026
- 2026AgentGuard: An Attribute-Based Access Control Framework for Tool-Use LLM-Based Agent27 May 2026
- 2026Governance by Construction for Generalist Agents20 May 2026
- 2026SafeHarness: Lifecycle-Integrated Security Architecture for LLM-based Agent Deployment15 Apr 2026
- 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration24 Mar 2026
- 2026Learning When to Act or Refuse: Guarding Agentic Reasoning Models for Safe Multi-Step Tool Use3 Mar 2026
- 2026MirrorGuard: Toward Secure Computer-Use Agents via Simulation-to-Real Reasoning Correction19 Jan 2026
- 2025When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?15 Oct 2025
- 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models4 Sep 2025
- 2025HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimization6 Aug 2025
- 2025Agentic Web: Weaving the Next Web with AI Agents28 Jul 2025
- 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 57
- 2025Agentic Web: Weaving the Next Web with AI Agentscited by 45
- 2026SafeHarness: Lifecycle-Integrated Security Architecture for LLM-based Agent Deploymentcited by 15
- 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 14
- 2025Agent Safety Alignment via Reinforcement Learningcited by 11
- 2025HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimizationcited by 10
- 2026MirrorGuard: Toward Secure Computer-Use Agents via Simulation-to-Real Reasoning Correctioncited by 5
- 2025When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?cited by 3
- 2026AgentGuard: An Attribute-Based Access Control Framework for Tool-Use LLM-Based Agentcited by 3
- 2026Governance by Construction for Generalist Agentscited by 1
- 2026AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agentscited by 1
- 2026MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learningcited by 0
- 2026AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents27 Sep 2026
- 2026MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning19 Sep 2026
- 2026AgentGuard: An Attribute-Based Access Control Framework for Tool-Use LLM-Based Agent27 May 2026
- 2026Governance by Construction for Generalist Agents20 May 2026
- 2026SafeHarness: Lifecycle-Integrated Security Architecture for LLM-based Agent Deployment15 Apr 2026
- 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration24 Mar 2026
- 2026MirrorGuard: Toward Secure Computer-Use Agents via Simulation-to-Real Reasoning Correction19 Jan 2026
- 2025When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?15 Oct 2025
- 2025HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimization6 Aug 2025
- 2025Agentic Web: Weaving the Next Web with AI Agents28 Jul 2025
- 2025Agent Safety Alignment via Reinforcement Learning11 Jul 2025
- 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasures24 Jun 2025
Similar, not cited
Close in wording, no citation either way
- 2026StepSafe: Learning Timely Intervention with Context-Aware Safety Reasoning
- 2025AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Models
- 2025SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agents
- 2026Safety Context Injection: Inference-Time Safety Alignment via Static Filtering and Agentic Analysis
- 2026Discovering Agentic Safety Specifications from 1-Bit Danger Signals
- 2025A Framework for Benchmarking and Aligning Task-Planning Safety in LLM-Based Embodied Agents
- 2026DRAFT: Task Decoupled Latent Reasoning for Agent Safety
- 2026THINKSAFE: Self-Generated Safety Alignment for Reasoning Models