# Platform for Situated Intelligence

arXiv: https://arxiv.org/abs/2103.15975 · HTML: https://mlsanity.com/paper/2103.15975/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 44 references that are in the map.
- First posted on arXiv 29 Mar 2021.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Fixing the train-test resolution discrepancy](https://mlsanity.com/paper/1906.06423/) (2019)
- [The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches](https://mlsanity.com/paper/1803.01164/) (2018)
- [The Microsoft 2017 Conversational Speech Recognition System](https://mlsanity.com/paper/1708.06073/) (2017)
- [English Conversational Telephone Speech Recognition by Humans and Machines](https://mlsanity.com/paper/1703.02136/) (2017)
- [On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems](https://mlsanity.com/paper/1611.08309/) (2016)

## Built on by (most cited first, 9 of 9)

- [LibreFace: An Open-Source Toolkit for Deep Facial Expression Analysis](https://mlsanity.com/paper/2308.10713/) (2023)
- [The AffectToolbox: Affect Analysis for Everyone](https://mlsanity.com/paper/2402.15195/) (2024)
- [Empathic Grounding: Explorations using Multimodal Interaction and Large Language Models with Conversational Agents](https://mlsanity.com/paper/2407.01824/) (2024)
- [ReactGenie: A Development Framework for Complex Multimodal Interactions Using Large Language Models](https://mlsanity.com/paper/2306.09649/) (2023)
- [PARSE-Ego4D: Personal Action Recommendation Suggestions for Egocentric Videos](https://mlsanity.com/paper/2407.09503/) (2024)
- [TOM: A Development Platform For Wearable Intelligent Assistants](https://mlsanity.com/paper/2407.15523/) (2024)
- [Enabling a Social Robot to Process Social Cues to Detect when to Help a User](https://mlsanity.com/paper/2110.11075/) (2021)
- [DISCOVER: A Data-driven Interactive System for Comprehensive Observation, Visualization, and ExploRation of Human Behaviour](https://mlsanity.com/paper/2407.13408/) (2024)
- [Dialogue Systems Engineering: A Survey and Future Directions](https://mlsanity.com/paper/2508.02279/) (2025)

## Abstract (arXiv)

We introduce Platform for Situated Intelligence, an open-source framework created to support the rapid development and study of multimodal, integrative-AI systems. The framework provides infrastructure for sensing, fusing, and making inferences from temporal streams of data across different modalities, a set of tools that enable visualization and debugging, and an ecosystem of components that encapsulate a variety of perception and processing technologies. These assets jointly provide the means for rapidly constructing and refining multimodal, integrative-AI systems, while retaining the efficiency and performance characteristics required for deployment in open-world settings.