# Human-AI Safety: A Descendant of Generative AI and Control Systems Safety

arXiv: https://arxiv.org/abs/2405.09794 · HTML: https://mlsanity.com/paper/2405.09794/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 121 references that are in the map.
- First posted on arXiv 16 May 2024.

## Builds on (most cited first, 30 of 63)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Aligning Text-to-Image Models using Human Feedback](https://mlsanity.com/paper/2302.12192/) (2023)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [Control Barrier Functions: Theory and Applications](https://mlsanity.com/paper/1903.11199/) (2019)
- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Measuring Progress on Scalable Oversight for Large Language Models](https://mlsanity.com/paper/2211.03540/) (2022)
- [Nash Learning from Human Feedback](https://mlsanity.com/paper/2312.00886/) (2023)
- [Hamilton-Jacobi Reachability: A Brief Overview and Recent Advances](https://mlsanity.com/paper/1709.07523/) (2017)
- [TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors](https://mlsanity.com/paper/2101.06557/) (2021)

## Built on by (most cited first, 7 of 7)

- [Generating Robot Constitutions & Benchmarks for Semantic Safety](https://mlsanity.com/paper/2503.08663/) (2025)
- [Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attacks](https://mlsanity.com/paper/2503.00187/) (2025)
- [Real-Time Out-of-Distribution Failure Prevention via Multi-Modal Reasoning](https://mlsanity.com/paper/2505.10547/) (2025)
- [ProgressGym: Alignment with a Millennium of Moral Progress](https://mlsanity.com/paper/2406.20087/) (2024)
- [Safe LLM-Controlled Robots with Formal Guarantees via Reachability Analysis](https://mlsanity.com/paper/2503.03911/) (2025)
- [RoboMD: Uncovering Robot Vulnerabilities through Semantic Potential Fields](https://mlsanity.com/paper/2412.02818/) (2024)
- [Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyond](https://mlsanity.com/paper/2410.18114/) (2024)

## Abstract (arXiv)

Artificial intelligence (AI) is interacting with people at an unprecedented scale, offering new avenues for immense positive impact, but also raising widespread concerns around the potential for individual and societal harm. Today, the predominant paradigm for human--AI safety focuses on fine-tuning the generative model's outputs to better agree with human-provided examples or feedback. In reality, however, the consequences of an AI model's outputs cannot be determined in isolation: they are tightly entangled with the responses and behavior of human users over time. In this paper, we distill key complementary lessons from AI safety and control systems safety, highlighting open challenges as well as key synergies between both fields. We then argue that meaningful safety assurances for advanced AI technologies require reasoning about how the feedback loop formed by AI outputs and human behavior may drive the interaction towards different outcomes. To this end, we introduce a unifying formalism to capture dynamic, safety-critical human--AI interactions and propose a concrete technical roadmap towards next-generation human-centered AI safety.