mlsanityData to 7 Oct 2026

AI Risk Management Should Incorporate Both Safety and Security

arXiv 2405.19524 ↗First posted 29 May 2024cs.CR cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

The exposure of security vulnerabilities in safety-aligned language models, e.g., susceptibility to adversarial attacks, has shed light on the intricate interplay between AI safety and AI security. Although the two disciplines now come together under the overarching goal of AI risk management, they have historically evolved separately, giving rise to differing perspectives. Therefore, in this paper, we advocate that stakeholders in AI risk management should be aware of the nuances, synergies, and interplay between safety and security, and unambiguously take into account the perspectives of both disciplines in order to devise mostly effective and holistic risk mitigation approaches. Unfortunately, this vision is often obfuscated, as the definitions of the basic concepts of "safety" and "security" themselves are often inconsistent and lack consensus across communities. With AI risk management being increasingly cross-disciplinary, this issue is particularly salient. In light of this conceptual challenge, we introduce a unified reference framework to clarify the differences and interplay between AI safety and AI security, aiming to facilitate a shared understanding and effective collaboration across communities.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 190 of its 285 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

190 of 285 references.

References · 190

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  9. 2013Intriguing properties of neural networkscited by 5,114
  10. 2017On Calibration of Modern Neural Networkscited by 4,860
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  7. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  11. 2016Concrete Problems in AI Safetycited by 1,582
  12. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  13. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  14. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  15. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  16. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  17. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  18. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  19. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  20. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  21. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  22. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  23. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  24. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  25. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  26. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  27. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  28. 2016Practical Black-Box Attacks against Machine Learningcited by 893
  29. 2018Robustness May Be at Odds with Accuracycited by 873
  30. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  31. 2021Membership Inference Attacks From First Principlescited by 839
  32. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  33. 2018How To Backdoor Federated Learningcited by 745
  34. 2023Extracting Training Data from Diffusion Modelscited by 723
  35. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  36. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  37. 2023A Watermark for Large Language Modelscited by 667
  38. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659

50 most cited of 190 All 190 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 7; 2026: 3.

17 papers.

Sort:

By area:

  1. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  2. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
  3. 2024Can Editing LLMs Inject Harm?cited by 22
  4. 2026When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use Agentscited by 13
  5. 2025PiCo: Jailbreaking Multimodal Large Language Models via Pictorial Code Contextualizationcited by 8
  6. 2024AEIOU: A Unified Defense Framework against NSFW Prompts in Text-to-Image Modelscited by 7
  7. 2025Trustworthy AI: Safety, Bias, and Privacy -- A Surveycited by 5
  8. 2025AI Safety vs. AI Security: Demystifying the Distinction and Boundariescited by 5
  9. 2025Limits of Safe AI Deployment: Differentiating Oversight and Controlcited by 3
  10. 2024Differentially Private Kernel Density Estimationcited by 2
  11. 2025AI Harmonics: a human-centric and harms severity-adaptive AI risk assessment frameworkcited by 2
  12. 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1
5 more
  1. 2025The Trust Calibration Maturity Model for Characterizing and Communicating Trustworthiness of AI Systemscited by 1
  2. 2024Data-Aware Training Quality Monitoring and Certification for Reliable Deep Learningcited by 0
  3. 2025Computational Safety for Generative AI: A Hypothesis Testing Perspectivecited by 0
  4. 2026Position: Retire the "Positive Backdoor" Label -- Secret Alignment Requires Strict and Systematic Evaluationcited by 0
  5. 2026Beyond Predictable Paths: Redefining AI Security Incident Reporting for Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
  2. 2025A Safety and Security Framework for Real-World Agentic Systems
  3. 2024Safeguarding AI Agents: Developing and Analyzing Safety Architectures
  4. 2024The AI risk repository: A meta-review, database, and taxonomy of risks from artificial intelligence
  5. 2026What Matters For Safety Alignment?
  6. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems
  7. 2024A Formal Framework for Assessing and Mitigating Emergent Security Risks in Generative AI Models: Bridging Theory and Dynamic Risk Mitigation
  8. 2024Large Language Model Safety: A Holistic Survey