AI Risk Management Should Incorporate Both Safety and Security
Not on the map. Why?
Abstract (arXiv)
The exposure of security vulnerabilities in safety-aligned language models, e.g., susceptibility to adversarial attacks, has shed light on the intricate interplay between AI safety and AI security. Although the two disciplines now come together under the overarching goal of AI risk management, they have historically evolved separately, giving rise to differing perspectives. Therefore, in this paper, we advocate that stakeholders in AI risk management should be aware of the nuances, synergies, and interplay between safety and security, and unambiguously take into account the perspectives of both disciplines in order to devise mostly effective and holistic risk mitigation approaches. Unfortunately, this vision is often obfuscated, as the definitions of the basic concepts of "safety" and "security" themselves are often inconsistent and lack consensus across communities. With AI risk management being increasingly cross-disciplinary, this issue is particularly salient. In light of this conceptual challenge, we introduce a unified reference framework to clarify the differences and interplay between AI safety and AI security, aiming to facilitate a shared understanding and effective collaboration across communities.
Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 190 of its 285 references that are in the map. First posted on arXiv 29 May 2024.
Builds on
190 of 285 references.
References · 190
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2014Explaining and Harnessing Adversarial Examplescited by 7,287
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2013Intriguing properties of neural networkscited by 5,114
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
38 more
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
- 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
- 2020Extracting Training Data from Large Language Modelscited by 2,144
- 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
- 2022Large Language Models Encode Clinical Knowledgecited by 1,807
- 2016Concrete Problems in AI Safetycited by 1,582
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
- 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
- 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
- 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
- 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
- 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
- 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
- 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
- 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
- 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
- 2016Practical Black-Box Attacks against Machine Learningcited by 893
- 2018Robustness May Be at Odds with Accuracycited by 873
- 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
- 2021Membership Inference Attacks From First Principlescited by 839
- 2022Quantifying Memorization Across Neural Language Modelscited by 755
- 2018How To Backdoor Federated Learningcited by 745
- 2023Extracting Training Data from Diffusion Modelscited by 723
- 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
- 2017Evasion Attacks against Machine Learning at Test Timecited by 686
- 2023A Watermark for Large Language Modelscited by 667
- 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
50 most cited of 190 All 190 (JSON)
Built on by
Citing papers per year: 2024: 7; 2025: 7; 2026: 3.
17 papers.
Sort:
By area:
- 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
- 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
- 2024Can Editing LLMs Inject Harm?cited by 22
- 2026When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use Agentscited by 13
- 2025PiCo: Jailbreaking Multimodal Large Language Models via Pictorial Code Contextualizationcited by 8
- 2024AEIOU: A Unified Defense Framework against NSFW Prompts in Text-to-Image Modelscited by 7
- 2025Trustworthy AI: Safety, Bias, and Privacy -- A Surveycited by 5
- 2025AI Safety vs. AI Security: Demystifying the Distinction and Boundariescited by 5
- 2025Limits of Safe AI Deployment: Differentiating Oversight and Controlcited by 3
- 2024Differentially Private Kernel Density Estimationcited by 2
- 2025AI Harmonics: a human-centric and harms severity-adaptive AI risk assessment frameworkcited by 2
- 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1
5 more
- 2025The Trust Calibration Maturity Model for Characterizing and Communicating Trustworthiness of AI Systemscited by 1
- 2024Data-Aware Training Quality Monitoring and Certification for Reliable Deep Learningcited by 0
- 2025Computational Safety for Generative AI: A Hypothesis Testing Perspectivecited by 0
- 2026Position: Retire the "Positive Backdoor" Label -- Secret Alignment Requires Strict and Systematic Evaluationcited by 0
- 2026Beyond Predictable Paths: Redefining AI Security Incident Reporting for Agentscited by 0
- 2026Beyond Predictable Paths: Redefining AI Security Incident Reporting for Agents21 Sep 2026
- 2026Position: Retire the "Positive Backdoor" Label -- Secret Alignment Requires Strict and Systematic Evaluation27 May 2026
- 2026When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use Agents9 Feb 2026
- 2025AI Harmonics: a human-centric and harms severity-adaptive AI risk assessment framework12 Sep 2025
- 2025Limits of Safe AI Deployment: Differentiating Oversight and Control4 Jul 2025
- 2025AI Safety vs. AI Security: Demystifying the Distinction and Boundaries21 Jun 2025
- 2025PiCo: Jailbreaking Multimodal Large Language Models via Pictorial Code Contextualization2 Apr 2025
- 2025Computational Safety for Generative AI: A Hypothesis Testing Perspective18 Feb 2025
- 2025Trustworthy AI: Safety, Bias, and Privacy -- A Survey11 Feb 2025
- 2025The Trust Calibration Maturity Model for Characterizing and Communicating Trustworthiness of AI Systems28 Jan 2025
- 2024AEIOU: A Unified Defense Framework against NSFW Prompts in Text-to-Image Models24 Dec 2024
- 2024On Evaluating the Durability of Safeguards for Open-Weight LLMs10 Dec 2024
5 more
- 2024Data-Aware Training Quality Monitoring and Certification for Reliable Deep Learning14 Oct 2024
- 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyond9 Oct 2024
- 2024Differentially Private Kernel Density Estimation3 Sep 2024
- 2024Can Editing LLMs Inject Harm?29 Jul 2024
- 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun 2024
- 2024Can Editing LLMs Inject Harm?cited by 22
- 2025Trustworthy AI: Safety, Bias, and Privacy -- A Surveycited by 5
- 2025Computational Safety for Generative AI: A Hypothesis Testing Perspectivecited by 0
- 2025Computational Safety for Generative AI: A Hypothesis Testing Perspective18 Feb 2025
- 2025Trustworthy AI: Safety, Bias, and Privacy -- A Survey11 Feb 2025
- 2024Can Editing LLMs Inject Harm?29 Jul 2024
- 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
- 2024AEIOU: A Unified Defense Framework against NSFW Prompts in Text-to-Image Modelscited by 7
- 2025AI Safety vs. AI Security: Demystifying the Distinction and Boundariescited by 5
- 2025Limits of Safe AI Deployment: Differentiating Oversight and Controlcited by 3
- 2024Differentially Private Kernel Density Estimationcited by 2
- 2025AI Harmonics: a human-centric and harms severity-adaptive AI risk assessment frameworkcited by 2
- 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1
- 2025The Trust Calibration Maturity Model for Characterizing and Communicating Trustworthiness of AI Systemscited by 1
- 2024Data-Aware Training Quality Monitoring and Certification for Reliable Deep Learningcited by 0
- 2026Beyond Predictable Paths: Redefining AI Security Incident Reporting for Agentscited by 0
- 2026Beyond Predictable Paths: Redefining AI Security Incident Reporting for Agents21 Sep 2026
- 2025AI Harmonics: a human-centric and harms severity-adaptive AI risk assessment framework12 Sep 2025
- 2025Limits of Safe AI Deployment: Differentiating Oversight and Control4 Jul 2025
- 2025AI Safety vs. AI Security: Demystifying the Distinction and Boundaries21 Jun 2025
- 2025The Trust Calibration Maturity Model for Characterizing and Communicating Trustworthiness of AI Systems28 Jan 2025
- 2024AEIOU: A Unified Defense Framework against NSFW Prompts in Text-to-Image Models24 Dec 2024
- 2024On Evaluating the Durability of Safeguards for Open-Weight LLMs10 Dec 2024
- 2024Data-Aware Training Quality Monitoring and Certification for Reliable Deep Learning14 Oct 2024
- 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyond9 Oct 2024
- 2024Differentially Private Kernel Density Estimation3 Sep 2024
Similar, not cited
Close in wording, no citation either way
- 2026The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- 2025A Safety and Security Framework for Real-World Agentic Systems
- 2024Safeguarding AI Agents: Developing and Analyzing Safety Architectures
- 2024The AI risk repository: A meta-review, database, and taxonomy of risks from artificial intelligence
- 2026What Matters For Safety Alignment?
- 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems
- 2024A Formal Framework for Assessing and Mitigating Emergent Security Risks in Generative AI Models: Bridging Theory and Dynamic Risk Mitigation
- 2024Large Language Model Safety: A Holistic Survey