mlsanityData to 7 Oct 2026

Position: Towards Bidirectional Human-AI Alignment

arXiv 2406.09264 ↗First posted 13 Jun 2024cs.HC cs.AI cs.CLcited by 15

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Recent advances in general-purpose AI underscore the urgent need to align AI systems with human goals and values. Yet, the lack of a clear, shared understanding of what constitutes "alignment" limits meaningful progress and cross-disciplinary collaboration. In this position paper, we argue that the research community should explicitly define and critically reflect on "alignment" to account for the bidirectional and dynamic relationship between humans and AI. Through a systematic review of over 400 papers spanning HCI, NLP, ML, and more, we examine how alignment is currently defined and operationalized. Building on this analysis, we introduce the Bidirectional Human-AI Alignment framework, which not only incorporates traditional efforts to align AI with human values but also introduces the critical, underexplored dimension of aligning humans with AI -- supporting cognitive, behavioral, and societal adaptation to rapidly advancing AI technologies. Our findings reveal significant gaps in current literature, especially in long-term interaction design, human value modeling, and mutual understanding. We conclude with three central challenges and actionable recommendations to guide future research toward more nuanced, reciprocal, and human-AI alignment approaches.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 280 of its 443 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

280 of 443 references.

Closest ancestors · 22

Same direction, newest first

  1. 2024AI Alignment with Changing and Influenceable Reward Functions28 May
  2. 2024Investigating Cultural Alignment of Large Language Models20 Feb
  3. 2024A Roadmap to Pluralistic Alignment7 Feb
  4. 2023MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Tasks30 Oct
  5. 2023Learning From Free-Text Human Feedback -- Collect New Datasets Or Extend Existing Ones?24 Oct
  6. 2023Compositional preference models for aligning LMs17 Oct
  7. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values11 Oct
  8. 2023Human Feedback is not Gold Standard28 Sep
  9. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models30 Aug
  10. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models23 Aug
  11. 2023Aligning Language Models to User Opinions24 May
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback22 May
10 more
  1. 2023Training Language Models with Language Feedback at Scale28 Mar
  2. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  3. 2022Fine-tuning language models to find agreement among humans with diverse preferences28 Nov
  4. 2022Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control10 Nov
  5. 2022InterFair: Debiasing with Natural Language Feedback for Fair Interpretable Predictions14 Oct
  6. 2022Towards Boosting the Open-Domain Chatbot with Human Feedback30 Aug
  7. 2022Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback5 Aug
  8. 2022Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback12 Apr
  9. 2021MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning30 Dec
  10. 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Data15 Sep

Other references · 258

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2016Concrete Problems in AI Safetycited by 1,582
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  9. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  10. 2023Whose Opinions Do Language Models Reflect?cited by 759
  11. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  12. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
38 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  3. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  4. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  5. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  6. 2020Artificial Intelligence, Values and Alignmentcited by 443
  7. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  8. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  9. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  10. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  11. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  12. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
  13. 2023Explainability for Large Language Models: A Surveycited by 331
  14. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  15. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  16. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  17. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  18. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  19. 2018Improving fairness in machine learning systems: What do industry practitioners need?cited by 262
  20. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  21. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 253
  22. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 251
  23. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  24. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  25. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  26. 2023Fine-tuning Language Models for Factualitycited by 219
  27. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  28. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  29. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  30. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  31. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  32. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 189
  33. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  34. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 168
  35. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 166
  36. 2023Generative Judge for Evaluating Alignmentcited by 163
  37. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 162
  38. 2023Co-Writing with Opinionated Language Models Affects Users' Viewscited by 145

50 most cited of 258 All 258 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 12.

15 papers.

Sort:

By area:

  1. 2025Moral Change or Noise? On Problems of Aligning AI With Temporally Unstable Human Feedbackcited by 5
  2. 2025Bridging Distribution Shift and AI Safety: Conceptual and Methodological Synergiescited by 2
  3. 2025DoubleAgents: Human-Agent Alignment in a Socially Embedded Workflowcited by 1
  4. 2026Auditing Student-AI Collaboration: A Case Study of Online Graduate CS Studentscited by 1
  5. 2026Deployment-Relevant Alignment Cannot Be Inferred from Model-Level Evaluation Alonecited by 1
  6. 2026Intentmaking and Sensemaking: Human Interaction with AI-Guided Mathematical Discoverycited by 1
  7. 2026Adaptive Prompt Elicitation for Text-to-Image Generationcited by 0
  8. 2026Intra-Fairness Dynamics: The Bias Spillover Effect in Targeted LLM Alignmentcited by 0
  9. 2026Rationalize: Shared Semantic Reasoning for Human-AI Alignmentcited by 0
  10. 2026What Do People Actually Want From AI? Mapping Preference Pluralitycited by 0
  11. 2026A Taxonomy of Conceptual Alignment in Human-Robot Dialoguecited by 0
  12. 2026Identifying Harm in Personalized, Generative AI Systems Requires User-Centered Auditing at the Interaction Levelcited by 0
3 more
  1. 2026An Investigation of the NeurIPS and ICML 2025 Position Trackscited by 0
  2. 2026AdaLens: Interactive Storyline for Monitoring and Steering Long-Running Agentic Data Analysiscited by 0
  3. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Human-AI Interaction Alignment: Designing, Evaluating, and Evolving Value-Centered AI For Reciprocal Human-AI Futures
  2. 2026Align AI to Dynamic Human-AI Workflows
  3. 2026Toward a Theory of Value in AI Alignment
  4. 2024ValueCompass: A Framework for Measuring Contextual Value Alignment Between Human and LLMs
  5. 2026Building Intelligent User Interfaces for Human-AI Alignment
  6. 2024Concept Alignment
  7. 2024The AI Alignment Paradox
  8. 2024A Moral Imperative: The Need for Continual Superalignment of Large Language Models