mlsanityData to 7 Oct 2026

Satori: Towards Proactive AR Assistant with Belief-Desire-Intention User Modeling

arXiv 2410.16668 ↗First posted 22 Oct 2024cs.HC cs.AIcited by 24

Not on the map. Why?

Abstract (arXiv)

Augmented Reality (AR) assistance is increasingly used for supporting users with physical tasks like assembly and cooking. However, most systems rely on reactive responses triggered by user input, overlooking rich contextual and user-specific information. To address this, we present Satori, a novel AR system that proactively guides users by modeling both -- their mental states and environmental contexts. Satori integrates the Belief-Desire-Intention (BDI) framework with the state-of-the-art multi-modal large language model (LLM) to deliver contextually appropriate guidance. Our system is designed based on two formative studies involving twelve experts. We evaluated the system with a sixteen within-subject study and found that Satori matches the performance of designer-created Wizard-of-Oz (WoZ) systems, without manual configurations or heuristics, thereby improving generalizability, reusability, and expanding the potential of AR assistance.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 20 of its 138 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

20 of 138 references.

References · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 245
  5. 2018Human Action Recognition and Prediction: A Surveycited by 226
  6. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  7. 2020UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2cited by 111
  8. 2019Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Contextcited by 99
  9. 2021Predicting the Future from First Person (Egocentric) Vision: A Surveycited by 36
  10. 2023COKE: A Cognitive Knowledge Graph for Machine Theory of Mindcited by 26
  11. 2021IntentVizor: Towards Generic Query Guided Interactive Video Summarizationcited by 18
  12. 2020Hierarchical Conditional Relation Networks for Multimodal Video Question Answeringcited by 13
8 more
  1. 2024OmniActions: Predicting Digital Actions in Response to Real-World Multimodal Sensory Inputs with LLMscited by 10
  2. 2024Your Co-Workers Matter: Evaluating Collaborative Capabilities of Language Models in Blocks Worldcited by 9
  3. 2024SIGMA: An Open-Source Interactive System for Mixed-Reality Task Assistance Researchcited by 8
  4. 2020Building Proactive Voice Assistants: When and How (not) to Interactcited by 7
  5. 2023iBall: Augmenting Basketball Videos with Gaze-moderated Embedded Visualizationscited by 6
  6. 2013Anticipatory Mobile Computing: A Survey of the State of the Art and Research Challengescited by 2
  7. 2016Model-Based Testing, Using Belief-Desire-Intentions Agents, of Control Code for Robots in Collaborative Human-Robot Interactionscited by 2
  8. 2023Training for Open-Ended Drilling through a Virtual Reality Simulationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 17.

24 papers.

Sort:

By area:

  1. 2025ProMemAssist: Exploring Timely Proactive Assistance Through Working Memory Modeling in Multi-Modal Wearable Devicescited by 13
  2. 2025ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agentscited by 9
  3. 2025ProAgent: Harnessing On-Demand Sensory Contexts for Proactive LLM Agent Systems in the Wildcited by 7
  4. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactionscited by 6
  5. 2025When Should Users Check? Modeling Confirmation Frequency in Multi-Step Agentic AI Taskscited by 6
  6. 2026FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoningcited by 5
  7. 2026VisionClaw: Always-On AI Agents through Smart Glassescited by 4
  8. 2026SpeechLess: Micro-utterance with Personalized Spatial Memory-aware Assistant in Everyday Augmented Realitycited by 3
  9. 2026Seeing Eye to Eye: Enabling Cognitive Alignment Through Shared First-Person Perspective in Human-AI Collaborationcited by 3
  10. 2026Verbalizing LLMs' assumptions to explain and control sycophancycited by 3
  11. 2026Pro2Assist: Continuous Step-aware Proactive Assistance with Multi-modal Egocentric Perception for Long-horizon Procedural Taskscited by 2
  12. 2026Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluationcited by 2
12 more
  1. 2025Generative Augmented Reality: Paradigms, Technologies, and Future Applicationscited by 1
  2. 2026Memento: Towards Proactive Visualization of Everyday Memories with Personal Wearable AR Assistantcited by 1
  3. 2026From Speech-to-Spatial: Grounding Utterances on A Live Shared View with Augmented Realitycited by 1
  4. 2025Seeing My Future: Predicting Situated Interaction Behavior in Virtual Realitycited by 0
  5. 2026UNSEEN: A Cross-Stack LLM Unlearning Defense against AR-LLM Social Engineering Attackscited by 0
  6. 2026PhySE: A Psychological Framework for Real-Time AR-LLM Social Engineering Attackscited by 0
  7. 2026TRAFA: Anticipating User Actions to Reduce Errors in Procedural Tasks with Predictive Feedbackcited by 0
  8. 2026Inverse Theory of Mind Modeling for Content Recommendation: From Web Browsing to Dynamic Intelligent Interfacescited by 0
  9. 2026From Where to How: Continuous 4D Interaction Forecasting from Egocentric Videocited by 0
  10. 2026AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalizationcited by 0
  11. 2026Generative Tutorial: Towards Live Contextualized Visual Instructions for Physical Taskscited by 0
  12. 2026A Benchmarking Framework for Context-aware XR Interfacescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Explainable OOHRI: Communicating Robot Capabilities and Limitations as Augmented Reality Affordances ↗
  2. 2025Designing Memory-Augmented AR Agents for Spatiotemporal Reasoning in Personalized Task Assistance
  3. 2025An Embodied AR Navigation Agent: Integrating BIM with Retrieval-Augmented Generation for Language Guidance
  4. 2025YETI (YET to Intervene) Proactive Interventions by Multimodal AI Agents in Augmented Reality Tasks
  5. 2025Sensible Agent: A Framework for Unobtrusive Interaction with Proactive AR Agents
  6. 2024Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Model
  7. 2025Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agents
  8. 2026Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos