mlsanityData to 7 Oct 2026

NatSGLD: A Dataset with Speech, Gesture, Logic, and Demonstration for Robot Learning in Natural Human-Robot Interaction

arXiv 2502.16718 ↗First posted 23 Feb 2025cs.RO cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent advances in multimodal Human-Robot Interaction (HRI) datasets emphasize the integration of speech and gestures, allowing robots to absorb explicit knowledge and tacit understanding. However, existing datasets primarily focus on elementary tasks like object pointing and pushing, limiting their applicability to complex domains. They prioritize simpler human command data but place less emphasis on training robots to correctly interpret tasks and respond appropriately. To address these gaps, we present the NatSGLD dataset, which was collected using a Wizard of Oz (WoZ) method, where participants interacted with a robot they believed to be autonomous. NatSGLD records humans' multimodal commands (speech and gestures), each paired with a demonstration trajectory and a Linear Temporal Logic (LTL) formula that provides a ground-truth interpretation of the commanded tasks. This dataset serves as a foundational resource for research at the intersection of HRI and machine learning. By providing multimodal inputs and detailed annotations, NatSGLD enables exploration in areas such as multimodal instruction following, plan recognition, and human-advisable reinforcement learning from demonstrations. We release the dataset and code under the MIT License at https://www.snehesh.com/natsgld/ to support future HRI research.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 88 references that are in the map. First posted on arXiv 23 Feb 2025.

Builds on

33 of 88 references.

References · 33

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  4. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  5. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  6. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  7. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  8. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  9. 2022L2CS-Net: Fine-Grained Gaze Estimation in Unconstrained Environmentscited by 65
  10. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Modelscited by 58
  11. 2022Inferring Rewards from Language in Contextcited by 50
  12. 2020The EMPATHIC Framework for Task Learning from Implicit Human Feedbackcited by 38
21 more
  1. 2021Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systemscited by 36
  2. 2020PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewardscited by 30
  3. 2020Learning a natural-language to LTL executable semantic parser for grounded roboticscited by 22
  4. 2020Inverse Reinforcement Learning with Natural Language Goalscited by 20
  5. 2022From Spot 2.0 to Spot 2.10: What's New?cited by 20
  6. 2024Real-World Offline Reinforcement Learning from Vision Language Model Feedbackcited by 20
  7. 2023Gesture-Informed Robot Assistance via Foundation Modelscited by 16
  8. 2018Robust Gesture-Based Communication for Underwater Human-Robot Interaction in the context of Search and Rescue Diver Missionscited by 14
  9. 2021Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Taskscited by 13
  10. 2018Discovering Underlying Plans Based on Shallow Modelscited by 9
  11. 2018Multi-modal Feedback for Affordance-driven Interactive Reinforcement Learningcited by 9
  12. 2019Improved Optical Flow for Gesture-based Human-robot Interactioncited by 8
  13. 2017Object Referring in Visual Scene with Spoken Languagecited by 7
  14. 2021Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learningcited by 5
  15. 2023FEVA: Fast Event Video Annotation Toolcited by 4
  16. 2023Towards Invertible Semantic-Preserving Embeddings of Logical Formulaecited by 3
  17. 2017Recognizing Plans by Learning Embeddings from Observed Action Distributionscited by 2
  18. 2017Multichannel Robot Speech Recognition Database: MChRSRcited by 2
  19. 2023Multimodal Reinforcement Learning for Robots Collaborating with Humanscited by 2
  20. 2024Vision Beyond Boundaries: An Initial Design Space of Domain-specific Large Vision Models in Human-robot Interactioncited by 2
  21. 2023Co-speech gestures for human-robot collaborationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Multimodal Human-Autonomous Agents Interaction Using Pre-Trained Language and Visual Foundation Models
  2. 2019Probabilistic Multimodal Modeling for Human-Robot Interaction Tasks
  3. 2022Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers
  4. 2024The Conversation is the Command: Interacting with Real-World Autonomous Robot Through Natural Language
  5. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalization
  6. 2022Interactive Language: Talking to Robots in Real Time
  7. 2024Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Models
  8. 2026RoboTalk: Learning Multi-Robot Communication and Coordination from Multimodal Demonstrations