mlsanityData to 7 Oct 2026

Fundamental Limitations of Alignment in Large Language Models

arXiv 2304.11082 ↗First posted 19 Apr 2023cs.CL cs.AIcited by 152

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

An important aspect in developing language models that interact with humans is aligning their behavior to be useful and unharmful for their human users. This is usually achieved by tuning the model in a way that enhances desired behaviors and inhibits undesired ones, a process referred to as alignment. In this paper, we propose a theoretical approach called Behavior Expectation Bounds (BEB) which allows us to formally investigate several inherent characteristics and limitations of alignment in large language models. Importantly, we prove that within the limits of this framework, for any behavior that has a finite probability of being exhibited by the model, there exist prompts that can trigger the model into outputting this behavior, with probability that increases with the length of the prompt. This implies that any alignment process that attenuates an undesired behavior but does not remove it altogether, is not safe against adversarial prompting attacks. Furthermore, our framework hints at the mechanism by which leading alignment approaches such as reinforcement learning from human feedback make the LLM prone to being prompted into the undesired behaviors. This theoretical result is being experimentally demonstrated in large scale by the so called contemporary "chatGPT jailbreaks", where adversarial users trick the LLM into breaking its alignment guardrails by triggering it into acting as a malicious persona. Our results expose fundamental limitations in alignment of LLMs and bring to the forefront the need to devise reliable mechanisms for ensuring AI safety.

Key facts. Cited by 152 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 55 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

31 of 55 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar
  2. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  12. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
17 more
  1. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  2. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  3. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  4. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  5. 2021Unsolved Problems in ML Safetycited by 291
  6. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  7. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  8. 2022Language Models as Agent Modelscited by 131
  9. 2023The Learnability of In-Context Learningcited by 126
  10. 2020A Mathematical Exploration of Why Language Models Help Solve Downstream Taskscited by 71
  11. 2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34
  12. 2023On the Provable Advantage of Unsupervised Pretrainingcited by 21
  13. 2023Advances in apparent conceptual physics reasoning in GPT-4cited by 12
  14. 2021Automatically Exposing Problems with Neural Dialog Modelscited by 5
  15. 2023Can Large Language Models Change User Preference Adversarially?cited by 5
  16. 2020On the Ethics of Building AI in a Responsible Mannercited by 3
  17. 2022Lower Bounds for the Total Variation Distance Given Means and Variances of Distributionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 48; 2024: 54; 2025: 28; 2026: 22.

152 papers.

Sort:

By area:

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  6. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  7. 2023Detecting Language Model Attacks with Perplexitycited by 411
  8. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  9. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  10. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  11. 2023Challenges and Applications of Large Language Modelscited by 251
  12. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
38 more
  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  2. 2023Playing repeated games with Large Language Modelscited by 213
  3. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  4. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  5. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  6. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  7. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  8. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  9. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  10. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 102
  11. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 102
  12. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  13. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  14. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  15. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakerscited by 91
  16. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  17. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 77
  18. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 77
  19. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  20. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  21. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  22. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  23. 2024Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoningcited by 62
  24. 2023Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Modelscited by 56
  25. 2024Virtual Personas for Language Models via an Anthology of Backstoriescited by 49
  26. 2023LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?cited by 47
  27. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 46
  28. 2023Large Language Models for Telecom: Forthcoming Impact on the Industrycited by 44
  29. 2024Evaluating Language Model Agency through Negotiationscited by 43
  30. 2024R2-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoningcited by 41
  31. 2024AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learningcited by 40
  32. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedbackcited by 38
  33. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 36
  34. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignmentcited by 35
  35. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  36. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  37. 2023Interactive Natural Language Processingcited by 31
  38. 2024Whispering Experts: Neural Interventions for Toxicity Mitigation in Language Modelscited by 31

50 most cited of 152 All 152 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Large Language Model Alignment: A Survey
  2. 2025LLM Reinforcement in Context
  3. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  4. 2024Can Reinforcement Learning Unlock the Hidden Dangers in Aligned Large Language Models?
  5. 2023The Alignment Problem in Context
  6. 2024The Real, the Better: Aligning Large Language Models with Online Human Behaviors
  7. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  8. 2025A methodological analysis of prompt perturbations and their effect on attack success rates