mlsanityData to 7 Oct 2026

Composing Pick-and-Place Tasks By Grounding Language

arXiv 2102.08094 ↗First posted 16 Feb 2021cs.RO cs.AI cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

Controlling robots to perform tasks via natural language is one of the most challenging topics in human-robot interaction. In this work, we present a robot system that follows unconstrained language instructions to pick and place arbitrary objects and effectively resolves ambiguities through dialogues. Our approach infers objects and their relationships from input images and language expressions and can place objects in accordance with the spatial relations expressed by the user. Unlike previous approaches, we consider grounding not only for the picking but also for the placement of everyday objects from language. Specifically, by grounding objects and their spatial relations, we allow specification of complex placement instructions, e.g. "place it behind the middle red bowl". Our results obtained using a real-world PR2 robot demonstrate the effectiveness of our method in understanding pick-and-place language instructions and sequentially composing them to solve tabletop manipulation tasks. Videos are available at http://speechrobot.cs.uni-freiburg.de

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 16 Feb 2021.

Builds on

21 of 33 references.

References · 21

  1. 2017Mask R-CNNcited by 7,259
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  4. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  5. 2016Generative Adversarial Text to Image Synthesiscited by 735
  6. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  7. 2015Neural Module Networkscited by 375
  8. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 237
  9. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  10. 2016Modeling Relationships in Referential Expressions with Compositional Modular Networkscited by 106
  11. 2016Shape Completion Enabled Robotic Graspingcited by 79
  12. 2016A Joint Speaker-Listener-Reinforcer Model for Referring Expressionscited by 74
9 more
  1. 2017Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructionscited by 61
  2. 2018Interactive Visual Grounding of Referring Expressions for Human-Robot Interactioncited by 55
  3. 2016High precision grasp pose detection in dense cluttercited by 33
  4. 2019Adversarial Skill Networks: Unsupervised Robot Skill Learning from Videocited by 17
  5. 2020Learning Object Placements For Relational Instructions by Hallucinating Scene Representationscited by 16
  6. 2019Self-supervised 3D Shape and Viewpoint Estimation from Single Images for Roboticscited by 10
  7. 2019Object Placement Planning and Optimization for Robot Manipulatorscited by 9
  8. 2020Hindsight for Foresight: Unsupervised Structured Dynamics Models from Physical Interactioncited by 8
  9. 2017Metric Learning for Generalizing Spatial Relations to New Objectscited by 5

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 6; 2023: 6; 2024: 4.

20 papers.

Sort:

By area:

  1. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  2. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  3. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  4. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  5. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  6. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  7. 2021INVIGORATE: Interactive Visual Grounding and Grasping in Cluttercited by 32
  8. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
  9. 2022Affordance Learning from Play for Sample-Efficient Policy Learningcited by 27
  10. 2023InsActor: Instruction-driven Physics-based Characterscited by 23
  11. 2021Semantically Grounded Object Matching for Robust Robotic Scene Rearrangementcited by 21
  12. 2024Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitationcited by 18
8 more
  1. 2023Learning 6-DoF Fine-grained Grasp Detection Based on Part Affordance Groundingcited by 17
  2. 2024CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissectioncited by 15
  3. 2022Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placementcited by 5
  4. 2023Object-centric Inference for Language Conditioned Placement: A Foundation Model based Approachcited by 4
  5. 2023GVCCI: Lifelong Learning of Visual Grounding for Language-Guided Robotic Manipulationcited by 4
  6. 2024SInViG: A Self-Evolving Interactive Visual Agent for Human-Robot Interactioncited by 2
  7. 2021Evaluating and Improving Interactions with Hazy Oraclescited by 1
  8. 2023PGA: Personalizing Grasping Agents with Single Human-Robot Interactioncited by 0