SciTrace: Trajectory-Aware Safety Reasoning for Scientific Discovery Agents
On the map: LLM reasoning → Reasoning for agents, planning and code
also in 1 more
Abstract (arXiv)
LLM-based scientific agents have shown strong capacity for autonomous research, yet their safety layers remain structurally divorced from core reasoning: they inspect pipeline outputs rather than shaping the deliberation that produces them. This separation opens two failure modes: safety signals accumulated at one stage are discarded before the next, and sequences of individually benign tool calls can compose into harmful outcomes that no single-step filter detects. To address these challenges, we introduce SciTrace, a framework that weaves safety reasoning into every stage of the scientific agent pipeline. SciTrace couples two complementary mechanisms: a textitSafety-Intrinsic Reasoning Loop (SIR) that maintains a cumulative risk state across the Thinker, Experimenter, Writer, and Reviewer stages through joint task-and-safety deliberation, and a textitCompositional Tool-Chain Verifier (CTV) that performs trajectory-aware safety checks before execution, catching risks that surface only across multi-step tool sequences. Evaluated on 240 high-risk research tasks and 120 tool-related risk tasks spanning six scientific domains, SciTrace achieves state-of-the-art (SOTA) safety among compared frameworks across four backbone models: it consistently improves tool call safety and adversarial robustness while preserving scientific output quality, and it uncovers 78.8% of the compositional tool-chain escapes that single-step monitors miss. The project website is available at https://opensciagent.github.io/SciTrace/.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 50 of its 54 references that are in the map. First posted on arXiv 6 Jun 2026.
Builds on
50 of 54 references.
Closest ancestors · 2
Same direction, newest first
- 2025Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?21 Feb
- 2024Qwen2.5-Coder Technical Report18 Sep
Other references · 48
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
36 more
- 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
- 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
- 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
- 2022Red Teaming Language Models with Language Modelscited by 1,026
- 2024DeepSeek-V3 Technical Reportcited by 811
- 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
- 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
- 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
- 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
- 2025Accelerating scientific discovery with Co-Scientistcited by 401
- 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
- 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
- 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
- 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
- 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
- 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
- 2023An Overview of Catastrophic AI Riskscited by 222
- 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
- 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 145
- 2025Red-Teaming LLM Multi-Agent Systems via Communication Attackscited by 122
- 2023Evil Geniuses: Delving into the Safety of LLM-based Agentscited by 106
- 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 89
- 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 72
- 2024ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectorscited by 51
- 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
- 2025AgentSafe: Safeguarding Large Language Model-based Multi-agent Systems via Hierarchical Data Managementcited by 38
- 2024ResearchTown: Simulator of Human Research Communitycited by 37
- 2025Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedbackcited by 25
- 2025SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agentscited by 16
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026POLAR: Ontology-Guided Risk Prevention for Tool-Calling LLM Agents
- 2026SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents
- 2026Enhancing Agent Safety Judgment: Controlled Benchmark Rewriting and Analogical Reasoning for Deceptive Out-of-Distribution Scenarios
- 2026Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification
- 2026ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis
- 2025SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agents
- 2026SCOUT: Synergizing Reasoning and Tool-Use for Computer-Use Safety
- 2026PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety