mlsanityData to 14 Oct 2026

Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction

arXiv 1806.03831 ↗First posted 11 Jun 2018cs.RO cs.CL cs.CVcited by 56

Not on the map. Why?

Abstract (arXiv)

This paper presents INGRESS, a robot system that follows human natural language instructions to pick and place everyday objects. The core issue here is the grounding of referring expressions: infer objects and their relationships from input images and language expressions. INGRESS allows for unconstrained object categories and unconstrained language expressions. Further, it asks questions to disambiguate referring expressions interactively. To achieve these, we take the approach of grounding by generation and propose a two-stage neural network model for grounding. The first stage uses a neural network to generate visual descriptions of objects, compares them with the input language expression, and identifies a set of candidate objects. The second stage uses another neural network to examine all pairwise relations between the candidates and infers the most likely referred object. The same neural networks are used for both grounding and question generation for disambiguation. Experiments show that INGRESS outperformed a state-of-the-art method on the RefCOCO dataset and in robot experiments with humans.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 12; 2022: 11; 2023: 14; 2024: 5; 2025: 9; 2026: 4.

56 papers.

Sort:

By area:

  1. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  2. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  3. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  4. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 138
  5. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 116
  6. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 107
  7. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  8. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  9. 2023PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulationcited by 56
  10. 2023Statler: State-Maintaining Language Models for Embodied Reasoningcited by 40
  11. 2023A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Cluttercited by 34
  12. 2023InsActor: Instruction-driven Physics-based Characterscited by 24
38 more
  1. 2021A Joint Network for Grasp Detection Conditioned on Natural Language Commandscited by 23
  2. 2023Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Cluttercited by 22
  3. 2025Multimodal Referring Segmentation: A Surveycited by 22
  4. 2021Composing Pick-and-Place Tasks By Grounding Languagecited by 20
  5. 2023Language-Conditioned Imitation Learning with Base Skill Priors under Unstructured Datacited by 20
  6. 2022Learning 6-DoF Object Poses to Grasp Category-level Objects by Language Instructionscited by 19
  7. 2021OCID-Ref: A 3D Robotic Dataset with Embodied Language for Clutter Scene Groundingcited by 16
  8. 2024CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissectioncited by 15
  9. 2021Two-stage Visual Cues Enhancement Network for Referring Image Segmentationcited by 13
  10. 2022Policy Adaptation from Foundation Model Feedbackcited by 10
  11. 2019Disentangled Relational Representations for Explaining and Learning from Demonstrationcited by 8
  12. 2025IRef-VLA: A Benchmark for Interactive Referential Grounding with Imperfect Language in 3D Scenescited by 8
  13. 2021Talk-to-Resolve: Combining scene understanding and spatial dialogue to resolve granular task ambiguity for a collocated robotcited by 7
  14. 2024HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Modelscited by 7
  15. 2021Language Understanding for Field and Service Robots in a Priori Unknown Environmentscited by 6
  16. 2022Incremental Object Grounding Using Scene Graphscited by 6
  17. 2023WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Modelcited by 6
  18. 2023Asking More Informative Questions for Grounded Retrievalcited by 6
  19. 2022DoRO: Disambiguation of referred object for embodied agentscited by 5
  20. 2022Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placementcited by 5
  21. 2024General-purpose Clothes Manipulation with Semantic Keypointscited by 5
  22. 2022Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attentioncited by 4
  23. 2022I Know What You Draw: Learning Grasp Detection Conditioned on a Few Freehand Sketchescited by 4
  24. 2023GVCCI: Lifelong Learning of Visual Grounding for Language-Guided Robotic Manipulationcited by 4
  25. 2024SparseGrasp: Robotic Grasping via 3D Semantic Gaussian Splatting from Sparse Multi-View RGB Imagescited by 3
  26. 2025Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Groundingcited by 3
  27. 2021Few-Shot Visual Grounding for Natural Human-Robot Interactioncited by 2
  28. 2021DASH: Modularized Human Manipulation Simulation with Vision and Language for Embodied AIcited by 2
  29. 2023PROGrasp: Pragmatic Human-Robot Communication for Object Graspingcited by 2
  30. 2024SInViG: A Self-Evolving Interactive Visual Agent for Human-Robot Interactioncited by 2
  31. 2025RAIDER: Tool-Equipped Large Language Model Agent for Robotic Action Issue Detection, Explanation and Recoverycited by 2
  32. 2026SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentationcited by 2
  33. 2021Evaluating and Improving Interactions with Hazy Oraclescited by 1
  34. 2022Enhancing Interpretability and Interactivity in Robot Manipulation: A Neurosymbolic Approachcited by 1
  35. 2023CARTIER: Cartographic lAnguage Reasoning Targeted at Instruction Execution for Robotscited by 1
  36. 2025Deformable Attentive Visual Enhancement for Referring Segmentation Using Vision-Language Modelcited by 1
  37. 2023PGA: Personalizing Grasping Agents with Single Human-Robot Interactioncited by 0
  38. 2023Towards Generalizable Referring Image Segmentation via Target Prompt and Visual Coherencecited by 0

50 most cited of 56 All 56 (JSON)