mlsanityData to 14 Oct 2026

Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placement

arXiv 2210.00215 ↗First posted 1 Oct 2022cs.RO cs.CV cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We present a new method, PARsing And visual GrOuNding (ParaGon), for grounding natural language in object placement tasks. Natural language generally describes objects and spatial relations with compositionality and ambiguity, two major obstacles to effective language grounding. For compositionality, ParaGon parses a language instruction into an object-centric graph representation to ground objects individually. For ambiguity, ParaGon uses a novel particle-based graph neural network to reason about object placements with uncertainty. Essentially, ParaGon integrates a parsing algorithm into a probabilistic, data-driven learning framework. It is fully differentiable and trained end-to-end from data for robustness against complex, ambiguous language input.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 26 of its 51 references that are in the map. First posted on arXiv 1 Oct 2022.

Builds on

26 of 51 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2017Mask R-CNNcited by 7,277
  6. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  7. 2017Deep Setscited by 1,421
  8. 2020Object-Centric Learning with Slot Attentioncited by 876
  9. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  10. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  11. 2020SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decompositioncited by 190
  12. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
14 more
  1. 2016Discriminative Embeddings of Latent Variable Models for Structured Datacited by 131
  2. 2019Attention Guided Graph Convolutional Networks for Relation Extractioncited by 82
  3. 2018Particle Filter Networks with Application to Visual Localizationcited by 65
  4. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 59
  5. 2018Interactive Visual Grounding of Referring Expressions for Human-Robot Interactioncited by 56
  6. 2019Particle Filter Recurrent Neural Networkscited by 36
  7. 2019Differentiable Algorithm Networks for Composable Robot Learningcited by 24
  8. 2021NViSII: A Scriptable Tool for Photorealistic Image Generationcited by 24
  9. 2018Scene Graph Parsing as Dependency Parsingcited by 21
  10. 2021Composing Pick-and-Place Tasks By Grounding Languagecited by 20
  11. 2020Towards Differentiable Resamplingcited by 19
  12. 2020Learning Object Placements For Relational Instructions by Hallucinating Scene Representationscited by 16
  13. 2017Robot-Initiated Specification Repair through Grounded Language Interactioncited by 2
  14. 2020Multiplicative Gaussian Particle Filtercited by 1

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2; 2025: 1.

5 papers.

  1. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 77
  2. 2024Enhancing the LLM-Based Robot Manipulation Through Human-Robot Collaborationcited by 19
  3. 2024LINGO-Space: Language-Conditioned Incremental Grounding for Spacecited by 8
  4. 2023"Tidy Up the Table": Grounding Common-sense Objective for Tabletop Object Rearrangementcited by 1
  5. 2025C2F-Space: Coarse-to-Fine Space Grounding for Spatial Instructions using Vision-Language Modelscited by 0