# Position: Towards Bidirectional Human-AI Alignment

arXiv: https://arxiv.org/abs/2406.09264 · HTML: https://mlsanity.com/paper/2406.09264/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 280 of its 443 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 280)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [Identifying the Risks of LM Agents with an LM-Emulated Sandbox](https://mlsanity.com/paper/2309.15817/) (2023)
- [RRHF: Rank Responses to Align Language Models with Human Feedback without tears](https://mlsanity.com/paper/2304.05302/) (2023)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Fine-Grained Human Feedback Gives Better Rewards for Language Model Training](https://mlsanity.com/paper/2306.01693/) (2023)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://mlsanity.com/paper/2312.15685/) (2023)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://mlsanity.com/paper/2305.03047/) (2023)
- [AlignScore: Evaluating Factual Consistency with a Unified Alignment Function](https://mlsanity.com/paper/2305.16739/) (2023)

## Built on by (most cited first, 18 of 18)

- [Moral Change or Noise? On Problems of Aligning AI With Temporally Unstable Human Feedback](https://mlsanity.com/paper/2511.10032/) (2025)
- [Bridging Distribution Shift and AI Safety: Conceptual and Methodological Synergies](https://mlsanity.com/paper/2505.22829/) (2025)
- [DoubleAgents: Human-Agent Alignment in a Socially Embedded Workflow](https://mlsanity.com/paper/2509.12626/) (2025)
- [Auditing Student-AI Collaboration: A Case Study of Online Graduate CS Students](https://mlsanity.com/paper/2601.08697/) (2026)
- [Intra-Fairness Dynamics: The Bias Spillover Effect in Targeted LLM Alignment](https://mlsanity.com/paper/2602.16438/) (2026)
- [What Did They Mean? How LLMs Resolve Ambiguous Social Situations across Perspectives and Roles](https://mlsanity.com/paper/2604.23942/) (2026)
- [Deployment-Relevant Alignment Cannot Be Inferred from Model-Level Evaluation Alone](https://mlsanity.com/paper/2605.04454/) (2026)
- [Intentmaking and Sensemaking: Human Interaction with AI-Guided Mathematical Discovery](https://mlsanity.com/paper/2605.05921/) (2026)
- [Adaptive Prompt Elicitation for Text-to-Image Generation](https://mlsanity.com/paper/2602.04713/) (2026)
- [Rationalize: Shared Semantic Reasoning for Human-AI Alignment](https://mlsanity.com/paper/2605.30632/) (2026)
- [What Do People Actually Want From AI? Mapping Preference Plurality](https://mlsanity.com/paper/2606.06674/) (2026)
- [A Taxonomy of Conceptual Alignment in Human-Robot Dialogue](https://mlsanity.com/paper/2606.22360/) (2026)
- [Identifying Harm in Personalized, Generative AI Systems Requires User-Centered Auditing at the Interaction Level](https://mlsanity.com/paper/2608.14692/) (2026)
- [An Investigation of the NeurIPS and ICML 2025 Position Tracks](https://mlsanity.com/paper/2608.16894/) (2026)
- [AdaLens: Interactive Storyline for Monitoring and Steering Long-Running Agentic Data Analysis](https://mlsanity.com/paper/2608.17834/) (2026)
- [AI Alignment through a Game-theoretic Lens: A Survey](https://mlsanity.com/paper/2608.27910/) (2026)
- [Provenance-Aware Belief Updating in Human-AI Feedback Loops](https://mlsanity.com/paper/2610.13008/) (2026)
- [Trust Hinders Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2610.15136/) (2026)

## Abstract (arXiv)

Recent advances in general-purpose AI underscore the urgent need to align AI systems with human goals and values. Yet, the lack of a clear, shared understanding of what constitutes "alignment" limits meaningful progress and cross-disciplinary collaboration. In this position paper, we argue that the research community should explicitly define and critically reflect on "alignment" to account for the bidirectional and dynamic relationship between humans and AI. Through a systematic review of over 400 papers spanning HCI, NLP, ML, and more, we examine how alignment is currently defined and operationalized. Building on this analysis, we introduce the Bidirectional Human-AI Alignment framework, which not only incorporates traditional efforts to align AI with human values but also introduces the critical, underexplored dimension of aligning humans with AI -- supporting cognitive, behavioral, and societal adaptation to rapidly advancing AI technologies. Our findings reveal significant gaps in current literature, especially in long-term interaction design, human value modeling, and mutual understanding. We conclude with three central challenges and actionable recommendations to guide future research toward more nuanced, reciprocal, and human-AI alignment approaches.