# NatSGLD: A Dataset with Speech, Gesture, Logic, and Demonstration for Robot Learning in Natural Human-Robot Interaction

arXiv: https://arxiv.org/abs/2502.16718 · HTML: https://mlsanity.com/paper/2502.16718/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 88 references that are in the map.
- First posted on arXiv 23 Feb 2025.

## Builds on (most cited first, 30 of 33)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021)
- [Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition](https://mlsanity.com/paper/2307.14535/) (2023)
- [Task and Motion Planning with Large Language Models for Object Rearrangement](https://mlsanity.com/paper/2303.06247/) (2023)
- [L2CS-Net: Fine-Grained Gaze Estimation in Unconstrained Environments](https://mlsanity.com/paper/2203.03339/) (2022)
- [Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models](https://mlsanity.com/paper/2211.11736/) (2022)
- [Inferring Rewards from Language in Context](https://mlsanity.com/paper/2204.02515/) (2022)
- [The EMPATHIC Framework for Task Learning from Implicit Human Feedback](https://mlsanity.com/paper/2009.13649/) (2020)
- [Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systems](https://mlsanity.com/paper/2109.09904/) (2021)
- [PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards](https://mlsanity.com/paper/2007.15543/) (2020)
- [Learning a natural-language to LTL executable semantic parser for grounded robotics](https://mlsanity.com/paper/2008.03277/) (2020)
- [From Spot 2.0 to Spot 2.10: What's New?](https://mlsanity.com/paper/2206.11366/) (2022)
- [Real-World Offline Reinforcement Learning from Vision Language Model Feedback](https://mlsanity.com/paper/2411.05273/) (2024)
- [Inverse Reinforcement Learning with Natural Language Goals](https://mlsanity.com/paper/2008.06924/) (2020)
- [Gesture-Informed Robot Assistance via Foundation Models](https://mlsanity.com/paper/2309.02721/) (2023)
- [Robust Gesture-Based Communication for Underwater Human-Robot Interaction in the context of Search and Rescue Diver Missions](https://mlsanity.com/paper/1810.07122/) (2018)
- [Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks](https://mlsanity.com/paper/2107.05825/) (2021)
- [Discovering Underlying Plans Based on Shallow Models](https://mlsanity.com/paper/1803.02208/) (2018)
- [Multi-modal Feedback for Affordance-driven Interactive Reinforcement Learning](https://mlsanity.com/paper/1807.09991/) (2018)
- [Improved Optical Flow for Gesture-based Human-robot Interaction](https://mlsanity.com/paper/1905.08685/) (2019)
- [Object Referring in Visual Scene with Spoken Language](https://mlsanity.com/paper/1711.03800/) (2017)
- [Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning](https://mlsanity.com/paper/2110.05286/) (2021)
- [FEVA: Fast Event Video Annotation Tool](https://mlsanity.com/paper/2301.00482/) (2023)
- [Multimodal Reinforcement Learning for Robots Collaborating with Humans](https://mlsanity.com/paper/2303.07265/) (2023)
- [Towards Invertible Semantic-Preserving Embeddings of Logical Formulae](https://mlsanity.com/paper/2305.03143/) (2023)
- [Recognizing Plans by Learning Embeddings from Observed Action Distributions](https://mlsanity.com/paper/1712.01949/) (2017)

## Abstract (arXiv)

Recent advances in multimodal Human-Robot Interaction (HRI) datasets emphasize the integration of speech and gestures, allowing robots to absorb explicit knowledge and tacit understanding. However, existing datasets primarily focus on elementary tasks like object pointing and pushing, limiting their applicability to complex domains. They prioritize simpler human command data but place less emphasis on training robots to correctly interpret tasks and respond appropriately. To address these gaps, we present the NatSGLD dataset, which was collected using a Wizard of Oz (WoZ) method, where participants interacted with a robot they believed to be autonomous. NatSGLD records humans' multimodal commands (speech and gestures), each paired with a demonstration trajectory and a Linear Temporal Logic (LTL) formula that provides a ground-truth interpretation of the commanded tasks. This dataset serves as a foundational resource for research at the intersection of HRI and machine learning. By providing multimodal inputs and detailed annotations, NatSGLD enables exploration in areas such as multimodal instruction following, plan recognition, and human-advisable reinforcement learning from demonstrations. We release the dataset and code under the MIT License at https://www.snehesh.com/natsgld/ to support future HRI research.