mlsanityData to 7 Oct 2026

Designing for Human-Agent Alignment: Understanding what humans want from their agents

arXiv 2404.04289 ↗First posted 4 Apr 2024cs.AI cs.HC cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Our ability to build autonomous agents that leverage Generative AI continues to increase by the day. As builders and users of such agents it is unclear what parameters we need to align on before the agents start performing tasks on our behalf. To discover these parameters, we ran a qualitative empirical research study about designing agents that can negotiate during a fictional yet relatable task of selling a camera online. We found that for an agent to perform the task successfully, humans/users and agents need to align over 6 dimensions: 1) Knowledge Schema Alignment 2) Autonomy and Agency Alignment 3) Operational Alignment and Training 4) Reputational Heuristics Alignment 5) Ethics Alignment and 6) Human Engagement Alignment. These empirical findings expand previous work related to process and specification alignment and the need for values and safety in Human-AI interactions. Subsequently we discuss three design directions for designers who are imagining a world filled with Human-Agent collaborations.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 16 of its 31 references that are in the map. First posted on arXiv 4 Apr 2024.

Builds on

16 of 31 references.

References · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2023Large Language Modelscited by 774
  5. 2020Artificial Intelligence, Values and Alignmentcited by 443
  6. 2018AI safety via debatecited by 384
  7. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  8. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  9. 2018Emergent Communication through Negotiationcited by 62
  10. 2023RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressionscited by 43
  11. 2023ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principlescited by 33
  12. 2023Interactive AI Alignment: Specification, Process, and Evaluation Alignmentcited by 22
4 more
  1. 2023Investigating How Practitioners Use Human-AI Guidelines: A Case Study on the People + AI Guidebookcited by 21
  2. 2023`It is currently hodgepodge'': Examining AI/ML Practitioners' Challenges during Co-production of Responsible AI Valuescited by 18
  3. 2021Emergent Communication under Competitioncited by 17
  4. 2016Effects of Sensemaking Translucence on Distributed Collaborative Analysiscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 3.

18 papers.

Sort:

By area:

  1. 2025Magentic-UI: Towards Human-in-the-loop Agentic Systemscited by 41
  2. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  3. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  4. 2025VeriLA: A Human-Centered Evaluation Framework for Interpretable Verification of LLM Agent Failurescited by 14
  5. 2025AI Chatbots for Mental Health: Values and Harms from Lived Experiences of Depressioncited by 13
  6. 2025Mapping Caregiver Needs to AI Chatbot Design: Strengths and Gaps in Mental Health Support for Alzheimer's and Dementia Caregiverscited by 13
  7. 2025Humanizing Machines: Rethinking LLM Anthropomorphism Through a Multi-Level Framework of Designcited by 11
  8. 2025From Conversation to Orchestration: HCI Challenges and Opportunities in Interactive Multi-Agentic Systemscited by 9
  9. 2025CoBRA: Programming Cognitive Bias in Social Agents Using Classic Social Science Experimentscited by 8
  10. 2025Distributional AGI Safetycited by 8
  11. 2026PrefIx: Understand and Adapt to User Preference in Human-Agent Interactioncited by 7
  12. 2025Attacks by Content: Automated Fact-checking is an AI Security Issuecited by 4
6 more
  1. 2026Plover: Steering GUI Agents through Plan-Centric Interactioncited by 2
  2. 2025RevTogether: Supporting Science Story Revision with Multiple AI Agentscited by 1
  3. 2025Customer Service Representative's Perception of the AI Assistant in an Organization's Call Centercited by 1
  4. 2026Alignment-Process-Outcome: Rethinking How AIs and Humans Collaboratecited by 1
  5. 2025Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agentscited by 0
  6. 2025Human-AI Interaction Alignment: Designing, Evaluating, and Evolving Value-Centered AI For Reciprocal Human-AI Futurescited by 0