# NatSGD: A Dataset with Speech, Gestures, and Demonstrations for Robot Learning in Natural Human-Robot Interaction

arXiv: https://arxiv.org/abs/2403.02274 · HTML: https://mlsanity.com/paper/2403.02274/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 86 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 25 of 25)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021)
- [Language-Conditioned Imitation Learning for Robot Manipulation Tasks](https://mlsanity.com/paper/2010.12083/) (2020)
- [Learning Speech-driven 3D Conversational Gestures from Video](https://mlsanity.com/paper/2102.06837/) (2021)
- [iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis](https://mlsanity.com/paper/2107.00285/) (2021)
- [Sim2Real Transfer for Reinforcement Learning without Dynamics Randomization](https://mlsanity.com/paper/2002.11635/) (2020)
- [A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020](https://mlsanity.com/paper/2102.11617/) (2021)
- [i-Sim2Real: Reinforcement Learning of Robotic Policies in Tight Human-Robot Interaction Loops](https://mlsanity.com/paper/2207.06572/) (2022)
- [Learning a natural-language to LTL executable semantic parser for grounded robotics](https://mlsanity.com/paper/2008.03277/) (2020)
- [From Spot 2.0 to Spot 2.10: What's New?](https://mlsanity.com/paper/2206.11366/) (2022)
- [Robust Gesture-Based Communication for Underwater Human-Robot Interaction in the context of Search and Rescue Diver Missions](https://mlsanity.com/paper/1810.07122/) (2018)
- [Multimodal analysis of the predictability of hand-gesture properties](https://mlsanity.com/paper/2108.05762/) (2021)
- [Discovering Underlying Plans Based on Shallow Models](https://mlsanity.com/paper/1803.02208/) (2018)
- [Improved Optical Flow for Gesture-based Human-robot Interaction](https://mlsanity.com/paper/1905.08685/) (2019)
- [Object Referring in Visual Scene with Spoken Language](https://mlsanity.com/paper/1711.03800/) (2017)
- [A Formal Analysis of Multimodal Referring Strategies Under Common Ground](https://mlsanity.com/paper/2003.07385/) (2020)
- [FEVA: Fast Event Video Annotation Tool](https://mlsanity.com/paper/2301.00482/) (2023)
- [Towards Invertible Semantic-Preserving Embeddings of Logical Formulae](https://mlsanity.com/paper/2305.03143/) (2023)
- [Recognizing Plans by Learning Embeddings from Observed Action Distributions](https://mlsanity.com/paper/1712.01949/) (2017)
- [Multichannel Robot Speech Recognition Database: MChRSR](https://mlsanity.com/paper/1801.00061/) (2017)
- [Reliable Attribute-Based Object Recognition Using High Predictive Value Classifiers](https://mlsanity.com/paper/1609.03619/) (2016)

## Built on by (most cited first, 4 of 4)

- [doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigation](https://mlsanity.com/paper/2412.05893/) (2024)
- [Exploring 3D Human Pose Estimation and Forecasting from the Robot's Perspective: The HARPER Dataset](https://mlsanity.com/paper/2403.14447/) (2024)
- [Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programming](https://mlsanity.com/paper/2409.11041/) (2024)
- [STARS: From Spatiotemporal Dynamics to Social Representations in Human-Robot Interaction](https://mlsanity.com/paper/2609.40245/) (2026)

## Abstract (arXiv)

Recent advancements in multimodal Human-Robot Interaction (HRI) datasets have highlighted the fusion of speech and gesture, expanding robots' capabilities to absorb explicit and implicit HRI insights. However, existing speech-gesture HRI datasets often focus on elementary tasks, like object pointing and pushing, revealing limitations in scaling to intricate domains and prioritizing human command data over robot behavior records. To bridge these gaps, we introduce NatSGD, a multimodal HRI dataset encompassing human commands through speech and gestures that are natural, synchronized with robot behavior demonstrations. NatSGD serves as a foundational resource at the intersection of machine learning and HRI research, and we demonstrate its effectiveness in training robots to understand tasks through multimodal human commands, emphasizing the significance of jointly considering speech and gestures. We have released our dataset, simulator, and code to facilitate future research in human-robot interaction system learning; access these resources at https://www.snehesh.com/natsgd/