mlsanityData to 7 Oct 2026

Risks of AI Scientists: Prioritizing Safeguarding Over Autonomy

arXiv 2402.04247 ↗First posted 6 Feb 2024cs.CY cs.AI cs.CLcited by 49

Not on the map. Why?

Cited by 20 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

AI scientists powered by large language models have demonstrated substantial promise in autonomously conducting experiments and facilitating scientific discoveries across various disciplines. While their capabilities are promising, these agents also introduce novel vulnerabilities that require careful consideration for safety. However, there has been limited comprehensive exploration of these vulnerabilities. This perspective examines vulnerabilities in AI scientists, shedding light on potential risks associated with their misuse, and emphasizing the need for safety measures. We begin by providing an overview of the potential risks inherent to AI scientists, taking into account user intent, the specific scientific domain, and their potential impact on the external environment. Then, we explore the underlying causes of these vulnerabilities and provide a scoping review of the limited existing works. Based on our analysis, we propose a triadic framework involving human regulation, agent alignment, and an understanding of environmental feedback (agent regulation) to mitigate these identified risks. Furthermore, we highlight the limitations and challenges associated with safeguarding AI scientists and advocate for the development of improved models, robust benchmarks, and comprehensive regulations.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 68 of its 110 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

68 of 110 references.

References · 68

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
38 more
  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  5. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  7. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  8. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  9. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  10. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  11. 2022Red Teaming Language Models with Language Modelscited by 983
  12. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  13. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  14. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  15. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  16. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  17. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  18. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  19. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  20. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  21. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  22. 2023Tool Learning with Foundation Modelscited by 302
  23. 2021Unsolved Problems in ML Safetycited by 291
  24. 2023Role-Play with Large Language Modelscited by 290
  25. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  26. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  27. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  28. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  29. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  30. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  31. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  32. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  33. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  34. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  35. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  36. 2023Agents: An Open-source Framework for Autonomous Language Agentscited by 116
  37. 2023GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Informationcited by 115
  38. 2023Evil Geniuses: Delving into the Safety of LLM-based Agentscited by 103

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 13; 2026: 15.

49 papers.

Sort:

By area:

  1. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  2. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 208
  3. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  4. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  5. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  6. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  7. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  8. 2024Safeguarding Large Language Models: A Surveycited by 73
  9. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  10. 2024Mora: Enabling Generalist Video Generation via A Multi-Agent Frameworkcited by 41
  11. 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Reviewcited by 30
  12. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
37 more
  1. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 26
  2. 2025The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systemscited by 24
  3. 2025ToolUniverse: An open platform for democratizing AI scientistscited by 22
  4. 2024Generating Interesting Scientific Ideas using Knowledge Graphs and LLMs: Evaluations with 100 Research Group Leaderscited by 21
  5. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimizationcited by 20
  6. 2024AutoStudio: Crafting Consistent Subjects in Multi-turn Interactive Image Generationcited by 19
  7. 2024AIGS: Generating Science from AI-Powered Automated Falsificationcited by 17
  8. 2024ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domaincited by 14
  9. 2024Defining and Detecting the Defects of the Large Language Model-based Autonomous Agentscited by 14
  10. 2025SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agentscited by 14
  11. 2024Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agentscited by 13
  12. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  13. 2025Agentic AI Systems Applied to tasks in Financial Services: Modeling and model risk management crewscited by 12
  14. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecturecited by 10
  15. 2025Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Papercited by 10
  16. 2025Characterizing Unintended Consequences in Human-GUI Agent Collaboration for Web Browsingcited by 7
  17. 2024FRACTURED-SORRY-Bench: Framework for Revealing Attacks in Conversational Turns Undermining Refusal Efficacy and Defenses over SORRY-Bench (Automated Multi-shot Jailbreaks)cited by 6
  18. 2024Rethinking Scientific Summarization Evaluation: Grounding Explainable Metrics on Facet-aware Benchmarkcited by 4
  19. 2026AI Coding Agents Can Reproduce Social Science Findingscited by 4
  20. 2025AI in a vat: Fundamental limits of efficient world modelling for agent sandboxing and interpretabilitycited by 3
  21. 2025Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Riskscited by 3
  22. 2026ForesightSafety Bench: A Frontier Risk Evaluation and Governance Framework towards Safe AIcited by 3
  23. 2026Discovery Foundation Models: Toward Open-Ended Discovery Intelligencecited by 3
  24. 2024Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systemscited by 2
  25. 2026Likelihood Hacking in Probabilistic Program Synthesiscited by 2
  26. 2026A Blind Spot in Alignment: Quantifying Biosecurity Risks in Large Language Modelscited by 2
  27. 2026MSP-LLM: A Unified Large Language Model Framework for Complete Material Synthesis Planningcited by 1
  28. 2026Toward Reliable, Safe, and Secure LLMs for Scientific Applicationscited by 1
  29. 2026Governing Reflective Human-AI Collaboration: A Framework for Epistemic Scaffolding and Traceable Reasoningcited by 1
  30. 2026SciTrace: Trajectory-Aware Safety Reasoning for Scientific Discovery Agentscited by 1
  31. 2026Locating Hidden Failures Makes Long-Horizon Agents More Reliablecited by 1
  32. 2025Project Rachel: Can an AI Become a Scholarly Author?cited by 0
  33. 2026Safe-SDL:Establishing Safety Boundaries and Control Mechanisms for AI-Driven Self-Driving Laboratoriescited by 0
  34. 2026From Overload to Insights: How AI Agents Can Support Scientists in Analyzing Complex Datacited by 0
  35. 2026SeekBrain: An Autonomous Multi-Agent System for Accelerating Neuroscience Discoverycited by 0
  36. 2026Towards a new paradigm of scientific discovery with socialized artificial intelligencecited by 0
  37. 2026Agentic self-driving microscopy benchmarks support qualification but do not necessarily generalize to unseen taskscited by 0