mlsanityData to 7 Oct 2026

INVIGORATE: Interactive Visual Grounding and Grasping in Clutter

arXiv 2108.11092 ↗First posted 25 Aug 2021cs.RO cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

This paper presents INVIGORATE, a robot system that interacts with human through natural language and grasps a specified object in clutter. The objects may occlude, obstruct, or even stack on top of one another. INVIGORATE embodies several challenges: (i) infer the target object among other occluding objects, from input language expressions and RGB images, (ii) infer object blocking relationships (OBRs) from the images, and (iii) synthesize a multi-step plan to ask questions that disambiguate the target object and to grasp it successfully. We train separate neural networks for object detection, for visual grounding, for question generation, and for OBR detection and grasping. They allow for unrestricted object categories and language expressions, subject to the training datasets. However, errors in visual perception and ambiguity in human languages are inevitable and negatively impact the robot's performance. To overcome these uncertainties, we build a partially observable Markov decision process (POMDP) that integrates the learned neural network modules. Through approximate POMDP planning, the robot tracks the history of observations and asks disambiguation questions in order to achieve a near-optimal sequence of actions that identify and grasp the target object. INVIGORATE combines the benefits of model-based POMDP planning and data-driven deep learning. Preliminary experiments with INVIGORATE on a Fetch robot show significant benefits of this integrated approach to object grasping in clutter with natural language interactions. A demonstration video is available at https://youtu.be/zYakh80SGcU.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 49 of its 86 references that are in the map. First posted on arXiv 25 Aug 2021.

Builds on

49 of 86 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  7. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  8. 2016Modeling Context in Referring Expressionscited by 1,054
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  11. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  12. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
37 more
  1. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  2. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  3. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  4. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  5. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  6. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 213
  7. 2018Differentiable MPC for End-to-end Planning and Controlcited by 178
  8. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  9. 2019Evaluating model calibration in classificationcited by 165
  10. 2016Value Iteration Networkscited by 158
  11. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  12. 2017Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matchingcited by 124
  13. 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 121
  14. 2013Deep Learning for Detecting Robotic Graspscited by 114
  15. 2013Data-Driven Grasp Synthesis - A Surveycited by 104
  16. 2020Referring Expression Comprehension: A Survey of Methods and Datasetscited by 78
  17. 2014Real-Time Grasp Detection Using Convolutional Neural Networkscited by 75
  18. 20196-DOF Grasping for Target-driven Object Manipulation in Cluttercited by 70
  19. 2020Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstructioncited by 66
  20. 2018Particle Filter Networks with Application to Visual Localizationcited by 64
  21. 2017Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructionscited by 61
  22. 2018Differentiable Particle Filters: End-to-End Learning with Algorithmic Priorscited by 58
  23. 2017QMDP-Net: Deep Learning for Planning under Partial Observabilitycited by 51
  24. 2016Dense Captioning with Joint Inference and Visual Contextcited by 40
  25. 2016Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measurescited by 36
  26. 2017Multi-Task Domain Adaptation for Deep Learning of Instance Grasping from Simulationcited by 35
  27. 2018Learning to Search with MCTSnetscited by 32
  28. 2019Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Cluttercited by 30
  29. 2019Differentiable Algorithm Networks for Composable Robot Learningcited by 24
  30. 2021A Joint Network for Grasp Detection Conditioned on Natural Language Commandscited by 23
  31. 2017End-to-End Learning of Semantic Graspingcited by 20
  32. 2021Composing Pick-and-Place Tasks By Grounding Languagecited by 20
  33. 2019A Deep Learning Approach to Grasping the Invisiblecited by 18
  34. 2020Visuomotor Mechanical Search: Learning to Retrieve Target Objects in Cluttercited by 16
  35. 2019LeTS-Drive: Driving in a Crowd by Learning from Tree Searchcited by 13
  36. 2018Visual Manipulation Relationship Networkcited by 10
  37. 2021Comparative evaluation of CNN architectures for Image Caption Generationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 9; 2024: 9; 2025: 3; 2026: 1.

32 papers.

Sort:

By area:

  1. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  2. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  3. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  4. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  5. 2023A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Cluttercited by 33
  6. 20253D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Taskscited by 30
  7. 2024Language-driven Grasp Detectioncited by 26
  8. 2023InsActor: Instruction-driven Physics-based Characterscited by 23
  9. 2021Semantically Grounded Object Matching for Robust Robotic Scene Rearrangementcited by 21
  10. 2024Reasoning Grasping via Multimodal Large Language Modelcited by 20
  11. 2023Learning 6-DoF Fine-grained Grasp Detection Based on Part Affordance Groundingcited by 17
  12. 2022Understanding Embodied Reference with Touch-Line Transformercited by 16
20 more
  1. 2024CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissectioncited by 15
  2. 2023Exploring Visual Pre-training for Robot Manipulation: Datasets, Models and Methodscited by 14
  3. 2024PreAfford: Universal Affordance-Based Pre-Grasping for Diverse Objects and Environmentscited by 11
  4. 2024NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulationcited by 10
  5. 2025IRef-VLA: A Benchmark for Interactive Referential Grounding with Imperfect Language in 3D Scenescited by 8
  6. 2021Talk-to-Resolve: Combining scene understanding and spatial dialogue to resolve granular task ambiguity for a collocated robotcited by 7
  7. 2024HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Modelscited by 7
  8. 2023WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Modelcited by 6
  9. 2024Robi Butler: Multimodal Remote Interaction with a Household Robot Assistantcited by 6
  10. 2022Toward Efficient Task Planning for Dual-Arm Tabletop Object Rearrangementcited by 5
  11. 2022DoRO: Disambiguation of referred object for embodied agentscited by 5
  12. 2025Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Promptingcited by 5
  13. 2023GVCCI: Lifelong Learning of Visual Grounding for Language-Guided Robotic Manipulationcited by 4
  14. 2024LLMs for Robotic Object Disambiguationcited by 4
  15. 2023PROGrasp: Pragmatic Human-Robot Communication for Object Graspingcited by 2
  16. 2023InViG: Benchmarking Interactive Visual Grounding with 500K Human-Robot Interactionscited by 2
  17. 2024SInViG: A Self-Evolving Interactive Visual Agent for Human-Robot Interactioncited by 2
  18. 2022Trust in Language Grounding: a new AI challenge for human-robot teamscited by 1
  19. 2023PGA: Personalizing Grasping Agents with Single Human-Robot Interactioncited by 0
  20. 2026Active Perception for Embodied Disambiguationcited by 0