mlsanityData to 7 Oct 2026

Human-AI Safety: A Descendant of Generative AI and Control Systems Safety

arXiv 2405.09794 ↗First posted 16 May 2024cs.AI cs.CY cs.SYcited by 7

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence (AI) is interacting with people at an unprecedented scale, offering new avenues for immense positive impact, but also raising widespread concerns around the potential for individual and societal harm. Today, the predominant paradigm for human--AI safety focuses on fine-tuning the generative model's outputs to better agree with human-provided examples or feedback. In reality, however, the consequences of an AI model's outputs cannot be determined in isolation: they are tightly entangled with the responses and behavior of human users over time. In this paper, we distill key complementary lessons from AI safety and control systems safety, highlighting open challenges as well as key synergies between both fields. We then argue that meaningful safety assurances for advanced AI technologies require reasoning about how the feedback loop formed by AI outputs and human behavior may drive the interaction towards different outcomes. To this end, we introduce a unifying formalism to capture dynamic, safety-critical human--AI interactions and propose a concrete technical roadmap towards next-generation human-centered AI safety.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 63 of its 121 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

63 of 121 references.

References · 63

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  10. 2018World Modelscited by 1,444
  11. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  12. 2022Red Teaming Language Models with Language Modelscited by 983
38 more
  1. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  2. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  5. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  6. 2017Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discoverycited by 566
  7. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  8. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  9. 2017Robust Adversarial Reinforcement Learningcited by 439
  10. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  11. 2021Adaptive Conformal Inference Under Distribution Shiftcited by 360
  12. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  13. 2019Control Barrier Functions: Theory and Applicationscited by 332
  14. 2021Unsolved Problems in ML Safetycited by 291
  15. 2023Nash Learning from Human Feedbackcited by 220
  16. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  17. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  18. 2022Guided Conditional Diffusion for Controllable Traffic Simulationcited by 189
  19. 2017Hamilton-Jacobi Reachability: A Brief Overview and Recent Advancescited by 186
  20. 2018The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigationcited by 159
  21. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 154
  22. 2022Safe Control with Learned Certificates: A Survey of Neural Lyapunov, Barrier, and Contraction methodscited by 148
  23. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  24. 2018Supervising strong learners by amplifying weak expertscited by 139
  25. 2020DROCC: Deep Robust One-Class Classificationcited by 121
  26. 2021Safe Nonlinear Control Using Robust Neural Lyapunov-Barrier Functionscited by 91
  27. 2020DeepReach: A Deep Learning Approach to High-Dimensional Reachabilitycited by 81
  28. 2021SimNet: Learning Reactive Self-driving Simulations from Real-world Observationscited by 77
  29. 2023The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systemscited by 77
  30. 2021Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificatescited by 71
  31. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 66
  32. 2022Practical Adversarial Multivalid Conformal Predictioncited by 65
  33. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  34. 2023Conformal Decision Theory: Safe Autonomous Decisions from Imperfect Predictionscited by 46
  35. 2023Can LLMs Follow Simple Rules?cited by 45
  36. 2014Reach-Avoid Problems with Time-Varying Dynamics, Targets and Constraintscited by 41
  37. 2023Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversationscited by 38
  38. 2019Preferences Implicit in the State of the Worldcited by 33

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 4.

7 papers.

  1. 2025Generating Robot Constitutions & Benchmarks for Semantic Safetycited by 28
  2. 2025Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attackscited by 20
  3. 2025Real-Time Out-of-Distribution Failure Prevention via Multi-Modal Reasoningcited by 10
  4. 2024ProgressGym: Alignment with a Millennium of Moral Progresscited by 9
  5. 2025Safe LLM-Controlled Robots with Formal Guarantees via Reachability Analysiscited by 7
  6. 2024RoboMD: Uncovering Robot Vulnerabilities through Semantic Potential Fieldscited by 5
  7. 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1