# Situated, Dynamic, and Subjective: Envisioning the Design of Theory-of-Mind-Enabled Everyday AI with Industry Practitioners

arXiv: https://arxiv.org/abs/2602.11342 · HTML: https://mlsanity.com/paper/2602.11342/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 93 references that are in the map.
- First posted on arXiv 11 Feb 2026.

## Builds on (most cited first, 17 of 17)

- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humans](https://mlsanity.com/paper/2001.04377/) (2020)
- [Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models](https://mlsanity.com/paper/2310.19619/) (2023)
- [On Benchmarking Human-Like Intelligence in Machines](https://mlsanity.com/paper/2502.20502/) (2025)
- [Re-evaluating Theory of Mind evaluation in large language models](https://mlsanity.com/paper/2502.21098/) (2025)
- [CX-ToM: Counterfactual Explanations with Theory-of-Mind for Enhancing Human Trust in Image Recognition Models](https://mlsanity.com/paper/2109.01401/) (2021)
- [Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience](https://mlsanity.com/paper/2302.10395/) (2023)
- [Prototyping with Prompts: Emerging Approaches and Challenges in Generative AI Design for Collaborative Software Teams](https://mlsanity.com/paper/2402.17721/) (2024)
- [Mutual Theory of Mind for Human-AI Communication](https://mlsanity.com/paper/2210.03842/) (2022)
- [Rethinking Theory of Mind Benchmarks for LLMs: Towards A User-Centered Perspective](https://mlsanity.com/paper/2504.10839/) (2025)
- [fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks](https://mlsanity.com/paper/2302.11703/) (2023)
- [Eliciting and Analysing Users' Envisioned Dialogues with Perfect Voice Assistants](https://mlsanity.com/paper/2102.13508/) (2021)
- [A Review on Machine Theory of Mind](https://mlsanity.com/paper/2303.11594/) (2023)
- [Sketching AI Concepts with Capabilities and Examples: AI Innovation in the Intensive Care Unit](https://mlsanity.com/paper/2402.13437/) (2024)
- ["What If Smart Homes Could See Our Homes?": Exploring DIY Smart Home Building Experiences with VLM-Based Camera Sensors](https://mlsanity.com/paper/2503.02816/) (2025)
- [Unremarkable to Remarkable AI Agent: Exploring Boundaries of Agent Intervention for Adults With and Without Cognitive Impairment](https://mlsanity.com/paper/2505.14872/) (2025)

## Built on by (most cited first, 1 of 1)

- [Disrupted Companionship: A Risk Assessment Framework and Cross-Platform Quantitative Analysis of Psychosocial Responses to AI Companion Disruptions](https://mlsanity.com/paper/2609.16907/) (2026)

## Abstract (arXiv)

Theory of Mind (ToM) -- the ability to infer what others are thinking (e.g., intentions) from observable cues -- is traditionally considered fundamental to human social interactions. This has sparked growing efforts in building and benchmarking AI's ToM capability, yet little is known about how such capability could translate into the design and experience of everyday user-facing AI products and services. We conducted 13 co-design sessions with 26 U.S.-based AI practitioners to envision, reflect, and distill design recommendations for ToM-enabled everyday AI products and services that are both future-looking and grounded in the realities of AI design and development practices. Analysis revealed three interrelated design recommendations: ToM-enabled AI should 1) be situated in the social context that shape users' mental states, 2) be responsive to the dynamic nature of mental states, and 3) be attuned to subjective individual differences. We surface design tensions within each recommendation that reveal a broader gap between practitioners' envisioned futures of ToM-enabled AI and the realities of current AI design and development practices. These findings point toward the need to move beyond static, inference-driven approach to ToM and toward designing ToM as a pervasive capability that supports continuous human-AI interaction loops.