# UNCOM: Zero-shot Context-Aware Command Understanding for Tabletop Scenarios

arXiv: https://arxiv.org/abs/2410.06355 · HTML: https://mlsanity.com/paper/2410.06355/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 24 references that are in the map.
- First posted on arXiv 8 Oct 2024.

## Builds on (most cited first, 5 of 5)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Adapting Everyday Manipulation Skills to Varied Scenarios](https://mlsanity.com/paper/1803.02743/) (2018)
- [Sharing Cognition: Human Gesture and Natural Language Grounding Based Planning and Navigation for Indoor Robots](https://mlsanity.com/paper/2108.06478/) (2021)

## Abstract (arXiv)

This paper presents UNCOM, a novel hybrid framework for interpreting natural human commands in tabletop scenarios. The system integrates multiple sources of information -- speech, gestures, and scene context -- to extract structured, actionable instructions for robots. Addressing the need for general-purpose human-robot interaction in domestic environments, UNCOM is designed for zero-shot operation, without reliance on predefined object models or training data specific to a given task. Using foundational and task-specific deep learning models, it allows out-of-the-box speech recognition, natural language understanding, gesture detection, and object segmentation. The modular architecture enhances transparency and explainability by explicitly parsing commands into object-action-target representations, enabling integration with symbolic robotic frameworks. We demonstrate the system in a TIAGo++ robot and provide an evaluation on a real-world data set of human-robot interaction scenarios; achieving an 82.39% success rate over our benchmark data set, highlighting the robustness of the system to diversity, noise, and communication ambiguity. The data set, evaluation scenarios, and the code are publicly available to support future research.