mlsanityData to 14 Oct 2026

Controlling risks of AI in chemical science with agents

arXiv 2312.06632 ↗First posted 11 Dec 2023cs.AIcited by 17

On the map: LLM agents and tool use → Web, search, and research agents

also in 1 more
Abstract (arXiv)

Artificial intelligence is rapidly advancing scientific discovery, but this progress carries risks of misuse, such as the creation of harmful substances, or circumvention of established regulations. In this paper, we first demonstrate the risks by highlighting real-world examples of AI misuse in chemical science, which underscore the need for effective safety alignment for these AI models. In response, we propose SciGuard, an agent-based guardrail that employs large language models, tools and external knowledge to assess and control risks in scientific AI interactions. For a fair comparison, we introduce a benchmark SciMT (Scientific Multi-Task) to assess both the safety and utility of different AI systems. SciGuard achieves a state-of-the-art harmlessness score on red-teaming queries, while maintaining high performance on benign tasks, without sacrificing scientific knowledge. Finally, we call for continued research and dialogue to ensure the safe deployment of AI in science.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 16 of its 75 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

16 of 75 references.

References · 16

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  9. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  10. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  11. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  12. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
4 more
  1. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 156
  2. 2023The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4cited by 112
  3. 2023Artificial intelligence and biological misuse: Differentiating risks of language models and biological design toolscited by 45
  4. 2023Will releasing the weights of future large language models grant widespread access to pandemic agents?cited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 8; 2026: 4.

17 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  2. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 93
  3. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  4. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 77
  5. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  6. 2025SciToolAgent: A Knowledge Graph-Driven Scientific Agent for Multi-Tool Integrationcited by 43
  7. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 42
  8. 2024"Not Aligned" is Not "Malicious": Being Careful about Hallucinations of Large Language Models' Jailbreakcited by 25
  9. 2024From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalitiescited by 19
  10. 2024SciSafeEval: A Comprehensive Benchmark for Safety Alignment of Large Language Models in Scientific Taskscited by 18
  11. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  12. 2025SoSBench: Benchmarking Safety Alignment on Six Scientific Domainscited by 12
5 more
  1. 2025Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibilitycited by 9
  2. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  3. 2026LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratoriescited by 3
  4. 2026Toward Reliable, Safe, and Secure LLMs for Scientific Applicationscited by 1
  5. 2026Molecular LLM Agents: From Architectural Design to Scientific Autonomycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents
  2. 2026Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
  3. 2025Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions
  4. 2026Toward Evaluation Frameworks for Multi-Agent Scientific AI Systems
  5. 2025Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?
  6. 2024LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labs
  7. 2026SciTrace: Trajectory-Aware Safety Reasoning for Scientific Discovery Agents
  8. 2026From Prompts to Protocols: An AI Agent for Laboratory Automation