# SIGMA: An Open-Source Interactive System for Mixed-Reality Task Assistance Research

arXiv: https://arxiv.org/abs/2405.13035 · HTML: https://mlsanity.com/paper/2405.13035/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 42 references that are in the map.
- First posted on arXiv 16 May 2024.

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models](https://mlsanity.com/paper/2308.01390/) (2023)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Language Is Not All You Need: Aligning Perception with Language Models](https://mlsanity.com/paper/2302.14045/) (2023)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World](https://mlsanity.com/paper/2309.17024/) (2023)
- [My View is the Best View: Procedure Learning from Egocentric Videos](https://mlsanity.com/paper/2207.10883/) (2022)
- [HoloLens 2 Research Mode as a Tool for Computer Vision Research](https://mlsanity.com/paper/2008.11239/) (2020)
- [Alexa Arena: A User-Centric Interactive Platform for Embodied AI](https://mlsanity.com/paper/2303.01586/) (2023)
- [Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?](https://mlsanity.com/paper/2311.00738/) (2023)
- [ARGUS: Visualization of AI-Assisted Task Guidance in AR](https://mlsanity.com/paper/2308.06246/) (2023)

## Built on by (most cited first, 8 of 8)

- [Satori: Towards Proactive AR Assistant with Belief-Desire-Intention User Modeling](https://mlsanity.com/paper/2410.16668/) (2024)
- [Explainable XR: Understanding User Behaviors of XR Environments using LLM-assisted Analytics Framework](https://mlsanity.com/paper/2501.13778/) (2025)
- [Transparent and Coherent Procedural Mistake Detection](https://mlsanity.com/paper/2412.11927/) (2024)
- [PARSE-Ego4D: Personal Action Recommendation Suggestions for Egocentric Videos](https://mlsanity.com/paper/2407.09503/) (2024)
- ["Is This It?": Towards Ecologically Valid Benchmarks for Situated Collaboration](https://mlsanity.com/paper/2409.10525/) (2024)
- [Grounding Task Assistance with Multimodal Cues from a Single Demonstration](https://mlsanity.com/paper/2505.01578/) (2025)
- [SigmaCollab: An Application-Driven Dataset for Physically Situated Collaboration](https://mlsanity.com/paper/2511.02560/) (2025)
- [Global Commander and Local Operative: A Dual-Agent Framework for Scene Navigation](https://mlsanity.com/paper/2602.18941/) (2026)

## Abstract (arXiv)

We introduce an open-source system called SIGMA (short for "Situated Interactive Guidance, Monitoring, and Assistance") as a platform for conducting research on task-assistive agents in mixed-reality scenarios. The system leverages the sensing and rendering affordances of a head-mounted mixed-reality device in conjunction with large language and vision models to guide users step by step through procedural tasks. We present the system's core capabilities, discuss its overall design and implementation, and outline directions for future research enabled by the system. SIGMA is easily extensible and provides a useful basis for future research at the intersection of mixed reality and AI. By open-sourcing an end-to-end implementation, we aim to lower the barrier to entry, accelerate research in this space, and chart a path towards community-driven end-to-end evaluation of large language, vision, and multimodal models in the context of real-world interactive applications.