# Challenges of Human-Aware AI Systems

arXiv: https://arxiv.org/abs/1910.07089 · HTML: https://mlsanity.com/paper/1910.07089/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2019.

## Built on by (most cited first, 13 of 13)

- [Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systems](https://mlsanity.com/paper/2109.09904/) (2021)
- [Challenges in Human-Agent Communication](https://mlsanity.com/paper/2412.10380/) (2024)
- [Integrating AI Planning with Natural Language Processing: A Combination of Explicit and Tacit Knowledge](https://mlsanity.com/paper/2202.07138/) (2022)
- [Simulation-Assisted Optimization for Large-Scale Evacuation Planning with Congestion-Dependent Delays](https://mlsanity.com/paper/2209.01535/) (2022)
- [Modeling Human Behavior Part I -- Learning and Belief Approaches](https://mlsanity.com/paper/2205.06485/) (2022)
- [Statistical relational learning and neuro-symbolic AI: what does first-order logic offer?](https://mlsanity.com/paper/2306.13660/) (2023)
- [A Risk Taxonomy and Reflection Tool for Large Language Model Adoption in Public Health](https://mlsanity.com/paper/2411.02594/) (2024)
- [AI-HRI Brings New Dimensions to Human-Aware Design for Human-Aware AI](https://mlsanity.com/paper/2210.11832/) (2022)
- [Roots and Requirements for Collaborative AIs](https://mlsanity.com/paper/2303.12040/) (2023)
- [Human-Modeling in Sequential Decision-Making: An Analysis through the Lens of Human-Aware AI](https://mlsanity.com/paper/2405.07773/) (2024)
- [Human-Aware Belief Revision: A Cognitively Inspired Framework for Explanation-Guided Revision of Human Models](https://mlsanity.com/paper/2405.19238/) (2024)
- [Counterfactual Explanations as Plans](https://mlsanity.com/paper/2502.09205/) (2025)
- [Improving LLM Performance Through Black-Box Online Tuning: A Case for Adding System Specs to Factsheets for Trusted AI](https://mlsanity.com/paper/2603.11340/) (2026)

## Abstract (arXiv)

From its inception, AI has had a rather ambivalent relationship to humans---swinging between their augmentation and replacement. Now, as AI technologies enter our everyday lives at an ever increasing pace, there is a greater need for AI systems to work synergistically with humans. To do this effectively, AI systems must pay more attention to aspects of intelligence that helped humans work with each other---including social intelligence. I will discuss the research challenges in designing such human-aware AI systems, including modeling the mental states of humans in the loop, recognizing their desires and intentions, providing proactive support, exhibiting explicable behavior, giving cogent explanations on demand, and engendering trust. I will survey the progress made so far on these challenges, and highlight some promising directions. I will also touch on the additional ethical quandaries that such systems pose. I will end by arguing that the quest for human-aware AI systems broadens the scope of AI enterprise, necessitates and facilitates true inter-disciplinary collaborations, and can go a long way towards increasing public acceptance of AI technologies.