mlsanityData to 14 Oct 2026

Foundational Challenges in Assuring Alignment and Safety of Large Language Models

arXiv 2404.09932 ↗First posted 15 Apr 2024cs.LG cs.AI cs.CLcited by 199

Not on the map. Why?

Cited by 47 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This work identifies 18 foundational challenges in assuring the alignment and safety of large language models (LLMs). These challenges are organized into three different categories: scientific understanding of LLMs, development and deployment methods, and sociotechnical challenges. Based on the identified challenges, we pose 200+ concrete research questions.

Key facts. Cited by 199 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Apr 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 77; 2025: 87; 2026: 35.

199 papers.

Sort:

By area:

  1. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  2. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 356
  3. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
  4. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  5. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  6. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  7. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  8. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 129
  9. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  10. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 125
  11. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 123
  12. 2024Emergent social conventions and collective bias in LLM populationscited by 123
38 more
  1. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  3. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 77
  4. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 73
  5. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 71
  6. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 67
  7. 2024Exploiting LLM Quantizationcited by 65
  8. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  9. 2024Yo'LLaVA: Your Personalized Language and Vision Assistantcited by 62
  10. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 61
  11. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 55
  12. 2025JBShield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulationcited by 54
  13. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 53
  14. 2024Virtual Personas for Language Models via an Anthology of Backstoriescited by 51
  15. 2024Scaling and renormalization in high-dimensional regressioncited by 50
  16. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 49
  17. 2024Persistent Pre-Training Poisoning of LLMscited by 48
  18. 2024Stress-Testing Capability Elicitation With Password-Locked Modelscited by 46
  19. 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
  20. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 45
  21. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 44
  22. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  23. 2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
  24. 2024Moral Alignment for LLM Agentscited by 39
  25. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 39
  26. 2025A Survey of Direct Preference Optimizationcited by 36
  27. 2024Agent Design Pattern Catalogue: A Collection of Architectural Patterns for Foundation Model based Agentscited by 35
  28. 2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
  29. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 34
  30. 2024Adversaries Can Misuse Combinations of Safe Modelscited by 32
  31. 2025Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedbackcited by 30
  32. 2024What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attackscited by 29
  33. 2025Mechanistic understanding and validation of large AI models with SemanticLenscited by 29
  34. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  35. 2025Mitigating Deceptive Alignment via Self-Monitoringcited by 29
  36. 2025Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steeringcited by 29
  37. 2024Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMscited by 28
  38. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28

50 most cited of 199 All 199 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
  2. 2024LongSafety: Enhance Safety for Long-Context LLMs
  3. 2025LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Models
  4. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  5. 2026LLM Safety Alignment in Low-Resource Languages: A Systematic Literature Review
  6. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  7. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixture
  8. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety