AI Safety in Generative AI Large Language Models: A Survey
Not on the map. Why?
Abstract (arXiv)
Large Language Model (LLMs) such as ChatGPT that exhibit generative AI capabilities are facing accelerated adoption and innovation. The increased presence of Generative AI (GAI) inevitably raises concerns about the risks and safety associated with these models. This article provides an up-to-date survey of recent trends in AI safety research of GAI-LLMs from a computer scientist's perspective: specific and technical. In this survey, we explore the background and motivation for the identified harms and risks in the context of LLMs being generative language models; our survey differentiates by emphasising the need for unified theories of the distinct safety challenges in the research development and applications of LLMs. We start our discussion with a concise introduction to the workings of LLMs, supported by relevant literature. Then we discuss earlier research that has pointed out the fundamental constraints of generative models, or lack of understanding thereof (e.g., performance and safety trade-offs as LLMs scale in number of parameters). We provide a sufficient coverage of LLM alignment -- delving into various approaches, contending methods and present challenges associated with aligning LLMs with human preferences. By highlighting the gaps in the literature and possible implementation oversights, our aim is to create a comprehensive analysis that provides insights for addressing AI safety in LLMs and encourages the development of aligned and secure models. We conclude our survey by discussing future directions of LLMs for AI safety, offering insights into ongoing research in this critical area.
Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 182 of its 254 references that are in the map. First posted on arXiv 6 Jul 2024.
Builds on
182 of 254 references.
References · 182
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
38 more
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
- 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
- 2020Learning to summarize from human feedbackcited by 2,562
- 2015Trust Region Policy Optimizationcited by 2,503
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
- 2020Extracting Training Data from Large Language Modelscited by 2,144
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
- 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
- 2020Big Bird: Transformers for Longer Sequencescited by 1,673
- 2016Concrete Problems in AI Safetycited by 1,582
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
- 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
- 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
- 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
- 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
- 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
- 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
50 most cited of 182 All 182 (JSON)
Built on by
Citing papers per year: 2024: 8; 2025: 18; 2026: 5.
31 papers.
Sort:
By area:
- 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 57
- 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
- 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
- 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
- 2025Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Modelscited by 19
- 2024SciSafeEval: A Comprehensive Benchmark for Safety Alignment of Large Language Models in Scientific Taskscited by 17
- 2025Compromising Honesty and Harmlessness in Language Models via Deception Attackscited by 17
- 2025GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detectioncited by 17
- 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
- 2025DrunkAgent: Stealthy Memory Corruption in LLM-Powered Recommender Agentscited by 12
- 2025JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Modelcited by 10
- 2024PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMscited by 6
19 more
- 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 5
- 2026State Contamination in Memory-Augmented LLM Agentscited by 4
- 2025On the Inevitability of Left-Leaning Political Bias in Aligned Language Modelscited by 3
- 2025Retrieval-Augmented Review Generation for Poisoning Recommender Systemscited by 3
- 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Modelscited by 3
- 2024Gen-AI for User Safety: A Surveycited by 2
- 2025Contextual Memory Intelligence -- A Foundational Paradigm for Human-AI Collaboration and Reflective Generative AI Systemscited by 2
- 2026The Hidden Puppet Master: Predicting Human Belief Change in Manipulative LLM Dialoguescited by 2
- 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1
- 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
- 2025From Symptoms to Systems: An Expert-Guided Approach to Understanding Risks of Generative AI for Eating Disorderscited by 1
- 2024A Formal Framework for Assessing and Mitigating Emergent Security Risks in Generative AI Models: Bridging Theory and Dynamic Risk Mitigationcited by 0
- 2024"Moralized" Multi-Step Jailbreak Prompts: Black-Box Testing of Guardrails in Large Language Models for Verbal Attackscited by 0
- 2025FLAME: Flexible LLM-Assisted Moderation Enginecited by 0
- 2025Computational Safety for Generative AI: A Hypothesis Testing Perspectivecited by 0
- 2025Limitations on Accurate, Trusted, Human-level Reasoningcited by 0
- 2026Responsible AI for General-Purpose Systems: Overview, Challenges, and A Path Forwardcited by 0
- 2026BarrierSteer: LLM Safety via Learning Barrier Steeringcited by 0
- 2026The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systemscited by 0
- 2026The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems21 Jul 2026
- 2026State Contamination in Memory-Augmented LLM Agents16 May 2026
- 2026The Hidden Puppet Master: Predicting Human Belief Change in Manipulative LLM Dialogues21 Mar 2026
- 2026BarrierSteer: LLM Safety via Learning Barrier Steering23 Feb 2026
- 2026Responsible AI for General-Purpose Systems: Overview, Challenges, and A Path Forward19 Jan 2026
- 2025From Symptoms to Systems: An Expert-Guided Approach to Understanding Risks of Generative AI for Eating Disorders4 Dec 2025
- 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models16 Nov 2025
- 2025Limitations on Accurate, Trusted, Human-level Reasoning25 Sep 2025
- 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems12 Sep 2025
- 2025Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Models2 Sep 2025
- 2025Retrieval-Augmented Review Generation for Poisoning Recommender Systems21 Aug 2025
- 2025Agentic Web: Weaving the Next Web with AI Agents28 Jul 2025
19 more
- 2025On the Inevitability of Left-Leaning Political Bias in Aligned Language Models21 Jul 2025
- 2025Pre-Trained Policy Discriminators are General Reward Models7 Jul 2025
- 2025Contextual Memory Intelligence -- A Foundational Paradigm for Human-AI Collaboration and Reflective Generative AI Systems28 May 2025
- 2025GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detection19 May 2025
- 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future12 Apr 2025
- 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languages6 Apr 2025
- 2025JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Model3 Apr 2025
- 2025DrunkAgent: Stealthy Memory Corruption in LLM-Powered Recommender Agents31 Mar 2025
- 2025Computational Safety for Generative AI: A Hypothesis Testing Perspective18 Feb 2025
- 2025FLAME: Flexible LLM-Assisted Moderation Engine13 Feb 2025
- 2025Compromising Honesty and Harmlessness in Language Models via Deception Attacks12 Feb 2025
- 2024LLMs are Also Effective Embedding Models: An In-depth Overview17 Dec 2024
- 2024"Moralized" Multi-Step Jailbreak Prompts: Black-Box Testing of Guardrails in Large Language Models for Verbal Attacks23 Nov 2024
- 2024Gen-AI for User Safety: A Survey10 Nov 2024
- 2024A Formal Framework for Assessing and Mitigating Emergent Security Risks in Generative AI Models: Bridging Theory and Dynamic Risk Mitigation15 Oct 2024
- 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyond9 Oct 2024
- 2024PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMs9 Oct 2024
- 2024SciSafeEval: A Comprehensive Benchmark for Safety Alignment of Large Language Models in Scientific Tasks2 Oct 2024
- 2024Recent Advances in Attack and Defense Approaches of Large Language Models5 Sep 2024
- 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
- 2025Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Modelscited by 19
- 2024SciSafeEval: A Comprehensive Benchmark for Safety Alignment of Large Language Models in Scientific Taskscited by 17
- 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
- 2025On the Inevitability of Left-Leaning Political Bias in Aligned Language Modelscited by 3
- 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Modelscited by 3
- 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models16 Nov 2025
- 2025Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Models2 Sep 2025
- 2025On the Inevitability of Left-Leaning Political Bias in Aligned Language Models21 Jul 2025
- 2025Pre-Trained Policy Discriminators are General Reward Models7 Jul 2025
- 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future12 Apr 2025
- 2024SciSafeEval: A Comprehensive Benchmark for Safety Alignment of Large Language Models in Scientific Tasks2 Oct 2024
- 2025GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detectioncited by 17
- 2024"Moralized" Multi-Step Jailbreak Prompts: Black-Box Testing of Guardrails in Large Language Models for Verbal Attackscited by 0
- 2025FLAME: Flexible LLM-Assisted Moderation Enginecited by 0
- 2025Computational Safety for Generative AI: A Hypothesis Testing Perspectivecited by 0
- 2026BarrierSteer: LLM Safety via Learning Barrier Steeringcited by 0
- 2026BarrierSteer: LLM Safety via Learning Barrier Steering23 Feb 2026
- 2025GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detection19 May 2025
- 2025Computational Safety for Generative AI: A Hypothesis Testing Perspective18 Feb 2025
- 2025FLAME: Flexible LLM-Assisted Moderation Engine13 Feb 2025
- 2024"Moralized" Multi-Step Jailbreak Prompts: Black-Box Testing of Guardrails in Large Language Models for Verbal Attacks23 Nov 2024
- 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
- 2025DrunkAgent: Stealthy Memory Corruption in LLM-Powered Recommender Agentscited by 12
- 2026State Contamination in Memory-Augmented LLM Agentscited by 4
- 2026State Contamination in Memory-Augmented LLM Agents16 May 2026
- 2025Agentic Web: Weaving the Next Web with AI Agents28 Jul 2025
- 2025DrunkAgent: Stealthy Memory Corruption in LLM-Powered Recommender Agents31 Mar 2025
- 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 57
- 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
- 2025Compromising Honesty and Harmlessness in Language Models via Deception Attackscited by 17
- 2024PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMscited by 6
- 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 5
- 2024Gen-AI for User Safety: A Surveycited by 2
- 2025Contextual Memory Intelligence -- A Foundational Paradigm for Human-AI Collaboration and Reflective Generative AI Systemscited by 2
- 2026The Hidden Puppet Master: Predicting Human Belief Change in Manipulative LLM Dialoguescited by 2
- 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1
- 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
- 2025From Symptoms to Systems: An Expert-Guided Approach to Understanding Risks of Generative AI for Eating Disorderscited by 1
- 2025Limitations on Accurate, Trusted, Human-level Reasoningcited by 0
12 most cited of 14 All 14 (JSON)
- 2026The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems21 Jul 2026
- 2026The Hidden Puppet Master: Predicting Human Belief Change in Manipulative LLM Dialogues21 Mar 2026
- 2026Responsible AI for General-Purpose Systems: Overview, Challenges, and A Path Forward19 Jan 2026
- 2025From Symptoms to Systems: An Expert-Guided Approach to Understanding Risks of Generative AI for Eating Disorders4 Dec 2025
- 2025Limitations on Accurate, Trusted, Human-level Reasoning25 Sep 2025
- 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems12 Sep 2025
- 2025Contextual Memory Intelligence -- A Foundational Paradigm for Human-AI Collaboration and Reflective Generative AI Systems28 May 2025
- 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languages6 Apr 2025
- 2025Compromising Honesty and Harmlessness in Language Models via Deception Attacks12 Feb 2025
- 2024LLMs are Also Effective Embedding Models: An In-depth Overview17 Dec 2024
- 2024Gen-AI for User Safety: A Survey10 Nov 2024
- 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyond9 Oct 2024
12 newest of 14 All 14 (JSON)
Similar, not cited
Close in wording, no citation either way
- 2024Large Language Model Safety: A Holistic Survey
- 2024SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types
- 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment
- 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts
- 2024Concept-Guided LLM Agents for Human-AI Safety Codesign
- 2024Current state of LLM Risks and AI Guardrails
- 2025Security Concerns for Large Language Models: A Survey
- 2025Challenges and Applications of Large Language Models: A Comparison of GPT and DeepSeek family of models