mlsanityData to 7 Oct 2026

COMPASS: Cognitive MCTS-Guided Process Alignment for Safe Search Agents

arXiv 2605.30838 ↗First posted 29 May 2026cs.AIcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

LLM-powered search agents enable multi-step reasoning and tool use. However, these capabilities introduce retrieval-induced safety degradation, as harmful intents may decompose into seemingly innocuous sub-queries that lead to unsafe outcomes. Existing alignment methods struggle to capture sparse safety signals and fail to supervise diverse violations across multi-step interactions. We propose COMPASS, a Cognitive MCTS-Guided Process Alignment framework designed to achieve robust safety alignment throughout the agent workflow while preserving general utility. COMPASS integrates cognitive tree exploration (CTE) to efficiently synthesize stealthy attack trajectories, and introspective step-wise alignment (ISA) to isolate risky intermediate actions for fine-grained process supervision. Empirical results show that COMPASS achieves a favorable safety-utility trade-off while requiring substantially less training data.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 35 of its 55 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

35 of 55 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents19 Oct
  2. 2025Safety Degradation in AI Agents20 May
  3. 2024RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent23 Jul

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  10. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
20 more
  1. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  3. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  4. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  5. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  6. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  7. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  8. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
  9. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  10. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  11. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 165
  12. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 120
  13. 2025Process Reward Models That Thinkcited by 118
  14. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  15. 2025Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learningcited by 68
  16. 2025s3: You Don't Need That Much Data to Train a Search Agent via RLcited by 41
  17. 2025Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Modelscited by 22
  18. 2025QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Searchcited by 20
  19. 2025Search Wisely: Mitigating Sub-optimal Agentic Searches By Reducing Uncertaintycited by 16
  20. 2025DecEx-RAG: Boosting Agentic Retrieval-Augmented Generation with Decision and Execution Optimization via Process Supervisioncited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Models
  2. 2026On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment
  3. 2025Agent Safety Alignment via Reinforcement Learning
  4. 2026AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
  5. 2026ToolAlignBench: Investigating Alignment Conflicts in Tool-Calling Enabled LLMs
  6. 2026When Safe Skills Collide: Measuring Compositional Risk in Agent Skill Ecosystems
  7. 2025Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correction
  8. 2026Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation