mlsanityData to 14 Oct 2026

SciTrace: Trajectory-Aware Safety Reasoning for Scientific Discovery Agents

arXiv 2606.08234 ↗First posted 6 Jun 2026cs.AIcited by 1

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

LLM-based scientific agents have shown strong capacity for autonomous research, yet their safety layers remain structurally divorced from core reasoning: they inspect pipeline outputs rather than shaping the deliberation that produces them. This separation opens two failure modes: safety signals accumulated at one stage are discarded before the next, and sequences of individually benign tool calls can compose into harmful outcomes that no single-step filter detects. To address these challenges, we introduce SciTrace, a framework that weaves safety reasoning into every stage of the scientific agent pipeline. SciTrace couples two complementary mechanisms: a textitSafety-Intrinsic Reasoning Loop (SIR) that maintains a cumulative risk state across the Thinker, Experimenter, Writer, and Reviewer stages through joint task-and-safety deliberation, and a textitCompositional Tool-Chain Verifier (CTV) that performs trajectory-aware safety checks before execution, catching risks that surface only across multi-step tool sequences. Evaluated on 240 high-risk research tasks and 120 tool-related risk tasks spanning six scientific domains, SciTrace achieves state-of-the-art (SOTA) safety among compared frameworks across four backbone models: it consistently improves tool call safety and adversarial robustness while preserving scientific output quality, and it uncovers 78.8% of the compositional tool-chain escapes that single-step monitors miss. The project website is available at https://opensciagent.github.io/SciTrace/.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 50 of its 54 references that are in the map. First posted on arXiv 6 Jun 2026.

Builds on

50 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?21 Feb
  2. 2024Qwen2.5-Coder Technical Report18 Sep

Other references · 48

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  11. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  12. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
36 more
  1. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  2. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  3. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  5. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  6. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  7. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  8. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  9. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  10. 2022Red Teaming Language Models with Language Modelscited by 1,026
  11. 2024DeepSeek-V3 Technical Reportcited by 811
  12. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  13. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  14. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  15. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  16. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
  17. 2025Accelerating scientific discovery with Co-Scientistcited by 401
  18. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
  19. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
  20. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  21. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  22. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
  23. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  24. 2023An Overview of Catastrophic AI Riskscited by 222
  25. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
  26. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 145
  27. 2025Red-Teaming LLM Multi-Agent Systems via Communication Attackscited by 122
  28. 2023Evil Geniuses: Delving into the Safety of LLM-based Agentscited by 106
  29. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 89
  30. 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 72
  31. 2024ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectorscited by 51
  32. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  33. 2025AgentSafe: Safeguarding Large Language Model-based Multi-agent Systems via Hierarchical Data Managementcited by 38
  34. 2024ResearchTown: Simulator of Human Research Communitycited by 37
  35. 2025Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedbackcited by 25
  36. 2025SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agentscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026POLAR: Ontology-Guided Risk Prevention for Tool-Calling LLM Agents
  2. 2026SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents
  3. 2026Enhancing Agent Safety Judgment: Controlled Benchmark Rewriting and Analogical Reasoning for Deceptive Out-of-Distribution Scenarios
  4. 2026Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification
  5. 2026ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis
  6. 2025SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agents
  7. 2026SCOUT: Synergizing Reasoning and Tool-Use for Computer-Use Safety
  8. 2026PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety