# Design Principles for Human-Agent Interaction

arXiv: https://arxiv.org/abs/2606.20630 · HTML: https://mlsanity.com/paper/2606.20630/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 64 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 7 of 7)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Culture-Based Explainable Human-Agent Deconfliction](https://mlsanity.com/paper/1911.10098/) (2019)
- [Human-Agent Coordination in Games under Incomplete Information via Multi-Step Intent](https://mlsanity.com/paper/2410.18242/) (2024)
- [Bootstrapping Linear Models for Fast Online Adaptation in Human-Agent Collaboration](https://mlsanity.com/paper/2404.10733/) (2024)
- [Social Identity in Human-Agent Interaction: A Primer](https://mlsanity.com/paper/2508.16609/) (2025)
- [Eliciting Spoken Interruptions to Inform Proactive Speech Agent Design](https://mlsanity.com/paper/2106.02077/) (2021)

## Abstract (arXiv)

AI agents are rapidly evolving into autonomous systems capable of sustained interaction, tool use, and long-term collaboration. Yet their real-world adoption remains limited, suggesting that the key barrier lies not only in technical capability but also in a lack of design knowledge for successful human-agent interaction. This position paper argues that AI agents should not be solely evaluated or deployed based on autonomous task capability alone; because agents interact with, adapt to, influence, and sometimes fail humans, human-agent interaction must be treated as a core design and evaluation target for agentic AI. We present 14 design principles that articulate the ideal human-agent relationship across four interaction stages: initially, during interaction, over time, and when things go wrong. We use these principles to evaluate nine agent systems to illustrate that these design principles can provide actionable guidance for AI design teams to systematically design and evaluate agents that are usable, trustworthy, and effective in real-world interactive settings.