mlsanityData to 14 Oct 2026

Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correction

arXiv 2505.11063 ↗First posted 16 May 2025cs.AI cs.CRcited by 14

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

LLM-based agents solve complex tasks through iterative reasoning, tool use, and environment interaction, where each intermediate thought directly shapes subsequent actions. Small deviations in these thoughts can therefore propagate into unsafe behaviors, yet existing guardrails typically operate only on final outputs or require intrusive model modifications. We introduce Thought-Aligner, a lightweight plug-in safety model that performs causal correction on unsafe thoughts before action execution, without altering the underlying agent. The corrected thoughts are fed back into the agent, steering its decision process and tool use toward safer trajectories. Because it operates solely at the thought level, Thought-Aligner is model-agnostic and can be integrated into diverse agent frameworks. We train Thought-Aligner via two-stage contrastive learning on paired safe and unsafe thoughts generated across ten risk scenarios. Experiments on diverse agent-safety benchmarks and six LLMs show that Thought-Aligner increases behavioral safety from about 50% without protection to around 90% on average, exceeding state-of-the-art guardrails by roughly 23%, while also improving helpfulness by about 5%. The method incurs low per-step latency and minimal overhead, enabling scalable and practical deployment. We publicly release Thought-Aligner-7B at https://huggingface.co/WhitzardAgent/Thought-Aligner-7B.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 59 of its 71 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

59 of 71 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning26 Mar
  2. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning13 Jun

Other references · 57

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  4. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  10. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  11. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  12. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
38 more
  1. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  2. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  3. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  4. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  5. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 538
  6. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  7. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  8. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 429
  9. 2023Language Models can Solve Computer Taskscited by 372
  10. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 369
  11. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  12. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  13. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 306
  14. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
  15. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  16. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  17. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  18. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 171
  19. 2023A Language Agent for Autonomous Drivingcited by 164
  20. 2023FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Designcited by 164
  21. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 151
  22. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  23. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 150
  24. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 137
  25. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 133
  26. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 131
  27. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 120
  28. 2024Attacking Vision-Language Computer Agents via Pop-upscited by 116
  29. 2024Aligner: Efficient Alignment by Learning to Correctcited by 98
  30. 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 96
  31. 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 92
  32. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agentscited by 90
  33. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 82
  34. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 81
  35. 2024EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Recordscited by 78
  36. 2024Imprompter: Tricking LLM Agents into Improper Tool Usecited by 68
  37. 2024AdvAgent: Controllable Blackbox Red-teaming on Web Agentscited by 59
  38. 2023Testing Language Model Agents Safely in the Wildcited by 46

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 8.

14 papers.

Sort:

By area:

Similar, not cited

Close in wording, no citation either way

  1. 2026StepSafe: Learning Timely Intervention with Context-Aware Safety Reasoning
  2. 2025AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Models
  3. 2025SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agents
  4. 2026Safety Context Injection: Inference-Time Safety Alignment via Static Filtering and Agentic Analysis
  5. 2026Discovering Agentic Safety Specifications from 1-Bit Danger Signals
  6. 2025A Framework for Benchmarking and Aligning Task-Planning Safety in LLM-Based Embodied Agents
  7. 2026DRAFT: Task Decoupled Latent Reasoning for Agent Safety
  8. 2026THINKSAFE: Self-Generated Safety Alignment for Reasoning Models