mlsanityData to 7 Oct 2026

Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions

arXiv 1710.06280 ↗First posted 17 Oct 2017cs.RO cs.CLcited by 61

Not on the map. Why?

Abstract (arXiv)

Comprehension of spoken natural language is an essential component for robots to communicate with human effectively. However, handling unconstrained spoken instructions is challenging due to (1) complex structures including a wide variety of expressions used in spoken language and (2) inherent ambiguity in interpretation of human instructions. In this paper, we propose the first comprehensive system that can handle unconstrained spoken language and is able to effectively resolve ambiguity in spoken instructions. Specifically, we integrate deep-learning-based object detection together with natural language processing technologies to handle unconstrained spoken instructions, and propose a method for robots to resolve instruction ambiguity through dialogue. Through our experiments on both a simulated environment as well as a physical industrial robot arm, we demonstrate the ability of our system to understand natural instructions from human operators effectively, and how higher success rates of the object picking task can be achieved through an interactive clarification process.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 15; 2023: 18; 2024: 8; 2025: 4; 2026: 2.

61 papers.

Sort:

By area:

  1. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  2. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  3. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  4. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  5. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  6. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 177
  7. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  8. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  9. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigationcited by 74
  10. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  11. 2023A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Cluttercited by 33
  12. 2021INVIGORATE: Interactive Visual Grounding and Grasping in Cluttercited by 32
38 more
  1. 2023Vision-Language Interpreter for Robot Task Planningcited by 29
  2. 2021A Joint Network for Grasp Detection Conditioned on Natural Language Commandscited by 23
  3. 2023Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Cluttercited by 22
  4. 2021Semantically Grounded Object Matching for Robust Robotic Scene Rearrangementcited by 21
  5. 2021Composing Pick-and-Place Tasks By Grounding Languagecited by 20
  6. 2022Learning 6-DoF Object Poses to Grasp Category-level Objects by Language Instructionscited by 19
  7. 2024CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissectioncited by 15
  8. 2023Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviorscited by 12
  9. 2022Policy Adaptation from Foundation Model Feedbackcited by 10
  10. 2021Target-dependent UNITER: A Transformer-Based Multimodal Language Comprehension Model for Domestic Service Robotscited by 7
  11. 2021Talk-to-Resolve: Combining scene understanding and spatial dialogue to resolve granular task ambiguity for a collocated robotcited by 7
  12. 2023Multimodal Diffusion Segmentation Model for Object Segmentation from Manipulation Instructionscited by 7
  13. 2024LINGO-Space: Language-Conditioned Incremental Grounding for Spacecited by 7
  14. 2022Incremental Object Grounding Using Scene Graphscited by 6
  15. 2023JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Modelscited by 6
  16. 2024Robi Butler: Multimodal Remote Interaction with a Household Robot Assistantcited by 6
  17. 2022DoRO: Disambiguation of referred object for embodied agentscited by 5
  18. 2023Switching Head-Tail Funnel UNITER for Dual Referring Expression Comprehension with Fetch-and-Carry Taskscited by 5
  19. 2023Learning-To-Rank Approach for Identifying Everyday Objects Using a Physical-World Search Enginecited by 5
  20. 2025Schema-Guided Scene-Graph Reasoning based on Multi-Agent Large Language Model Systemcited by 5
  21. 2021Case Relation Transformer: A Crossmodal Language Generation Model for Fetching Instructionscited by 4
  22. 2022Language Model-Based Paired Variational Autoencoders for Robotic Language Learningcited by 4
  23. 2022I Know What You Draw: Learning Grasp Detection Conditioned on a Few Freehand Sketchescited by 4
  24. 2023GVCCI: Lifelong Learning of Visual Grounding for Language-Guided Robotic Manipulationcited by 4
  25. 2022Music-driven Dance Regeneration with Controllable Key Pose Constraintscited by 3
  26. 2023Natural Language Instructions for Intuitive Human Interaction with Robotic Assistants in Field Construction Workcited by 3
  27. 2024DM2RM: Dual-Mode Multimodal Ranking for Target Objects and Receptacles Based on Open-Vocabulary Instructionscited by 3
  28. 2024Task Success Prediction for Open-Vocabulary Manipulation Based on Multi-Level Aligned Representationscited by 3
  29. 2024SparseGrasp: Robotic Grasping via 3D Semantic Gaussian Splatting from Sparse Multi-View RGB Imagescited by 3
  30. 2021Unsupervised Lexical Acquisition of Relative Spatial Concepts Using Spoken User Utterancescited by 2
  31. 2021DASH: Modularized Human Manipulation Simulation with Vision and Language for Embodied AIcited by 2
  32. 2022Sim-To-Real Transfer of Visual Grounding for Human-Aided Ambiguity Resolutioncited by 2
  33. 2023PROGrasp: Pragmatic Human-Robot Communication for Object Graspingcited by 2
  34. 2023InViG: Benchmarking Interactive Visual Grounding with 500K Human-Robot Interactionscited by 2
  35. 2025RAIDER: Tool-Equipped Large Language Model Agent for Robotic Action Issue Detection, Explanation and Recoverycited by 2
  36. 2021Neural Variational Learning for Grounded Language Acquisitioncited by 1
  37. 2021Evaluating and Improving Interactions with Hazy Oraclescited by 1
  38. 2022Learning Flexible Translation between Robot Actions and Language Descriptionscited by 1

50 most cited of 61 All 61 (JSON)