UserHarness: Harnessing User Minds for Stronger Agent Theory-of-Mind
Not on the map. Why?
Abstract (arXiv)
Understanding what a user believes and intends is central to building effective agent assistants. This ability is often evaluated through Theory-of-Mind (ToM) tasks, where success requires reasoning from the user's perspective. However, many existing approaches address ToM with complex pipelines that model behavior indirectly, without explicitly reconstructing the user's mental state. This misses the core structure of the problem: users act based on their beliefs, which are updated through observations of the environment; beliefs and intentions jointly determine actions, which in turn change the environment; and social reasoning often requires nested beliefs about what others believe or intend. We propose UserHarness, a simple framework that reframes ToM reasoning as explicit user-mind reconstruction. UserHarness decomposes the user's mental state, its relation to the external environment, and the actions that follow from it, enabling agents to track what the user observes, believes, intends, and does. Across five benchmarks, UserHarness reaches up to 95.94% macro accuracy, improving over existing inference methods by more than 15% relative and over the strongest prompt-only harness by about 20% relative. These results suggest that robust user understanding requires reasoning from the roots of the user's mind, positioning user harnessing as a promising foundation for more adaptive future assistants.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 18 of its 36 references that are in the map. First posted on arXiv 26 May 2026.
Builds on
18 of 36 references.
References · 18
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,953
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,335
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,154
- 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 290
- 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
- 2023Evaluating Large Language Models in Theory of Mind Taskscited by 218
- 2023Understanding Social Reasoning in Language Models with Language Modelscited by 167
- 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 162
- 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 162
- 2023Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Trackercited by 95
- 2023Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilitiescited by 86
- 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 85
6 more
- 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 71
- 2024MuMA-ToM: Multi-modal Multi-Agent Theory of Mindcited by 51
- 2024Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoningcited by 32
- 2025Hypothesis-Driven Theory-of-Mind Reasoning for Large Language Modelscited by 29
- 2025AutoToM: Scaling Model-based Mental Inference via Automated Agent Modelingcited by 23
- 2024TimeToM: Temporal Space is the Key to Unlocking the Door of Large Language Models' Theory-of-Mindcited by 20
Built on by
Citing papers per year: 2026: 1.
1 paper.