mlsanityData to 7 Oct 2026

StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects

arXiv 2110.10189 ↗First posted 19 Oct 2021cs.RO cs.AI cs.CLcited by 57

Not on the map. Why?

Abstract (arXiv)

Geometric organization of objects into semantically meaningful arrangements pervades the built world. As such, assistive robots operating in warehouses, offices, and homes would greatly benefit from the ability to recognize and rearrange objects into these semantically meaningful structures. To be useful, these robots must contend with previously unseen objects and receive instructions without significant programming. While previous works have examined recognizing pairwise semantic relations and sequential manipulation to change these simple relations none have shown the ability to arrange objects into complex structures such as circles or table settings. To address this problem we propose a novel transformer-based neural network, StructFormer, which takes as input a partial-view point cloud of the current object arrangement and a structured language command encoding the desired object configuration. We show through rigorous experiments that StructFormer enables a physical robot to rearrange novel objects into semantically meaningful structures with multi-object relational constraints inferred from the language command.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 37 of its 63 references that are in the map. First posted on arXiv 19 Oct 2021.

Builds on

37 of 63 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  3. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  7. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  8. 2020PCT: Point cloud transformercited by 730
  9. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  10. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  11. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 237
  12. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 225
25 more
  1. 2018Measuring abstract reasoning in neural networkscited by 165
  2. 2020SceneFormer: Indoor Scene Generation with Transformerscited by 154
  3. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  4. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  5. 2019PQ-NET: A Generative Part Seq2Seq Network for 3D Shapescited by 118
  6. 2020ACRONYM: A Large-Scale Grasp Dataset Based on Simulationcited by 115
  7. 2020Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphscited by 83
  8. 2019StructureNet: Hierarchical Graph Networks for 3D Shape Generationcited by 59
  9. 2021SORNet: Spatial Object-Centric Representations for Sequential Manipulationcited by 48
  10. 2021NeRP: Neural Rearrangement Planning for Unknown Objectscited by 33
  11. 2017GRASS: Generative Recursive Autoencoders for Shape Structurescited by 24
  12. 2021NViSII: A Scriptable Tool for Photorealistic Image Generationcited by 24
  13. 2017Representation Learning for Grounded Spatial Reasoningcited by 23
  14. 2021Predicting Stable Configurations for Semantic Placement of Novel Objectscited by 23
  15. 2019Prospection: Interpretable Plans From Language By Predicting the Futurecited by 21
  16. 2020Transferable Task Execution from Pixels through Deep Planning Domain Learningcited by 17
  17. 2020Learning Object Placements For Relational Instructions by Hallucinating Scene Representationscited by 16
  18. 2020Spatial Reasoning from Natural Language Instructions for Robot Manipulationcited by 16
  19. 2017Learning Interpretable Spatial Operations in a Rich 3D Blocks Worldcited by 15
  20. 2020Transformation Driven Visual Reasoningcited by 14
  21. 2020SHOP-VRB: A Visual Reasoning Benchmark for Object Perceptioncited by 13
  22. 2019Learning to Manipulate Object Collections Using Grounded State Representationscited by 11
  23. 2019Disentangled Relational Representations for Explaining and Learning from Demonstrationcited by 8
  24. 2017Metric Learning for Generalizing Spatial Relations to New Objectscited by 5
  25. 2021Optimizing Hospital Room Layout to Reduce the Risk of Patient Fallscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 13; 2023: 22; 2024: 14; 2025: 4; 2026: 3.

57 papers.

Sort:

By area:

  1. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  2. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  3. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  4. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  5. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  6. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  7. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  8. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 135
  9. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 105
  10. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  11. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  12. 2023LEGO-Net: Learning Regular Rearrangements of Objects in Roomscited by 75
38 more
  1. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  2. 2023OBJECT 3DIT: Language-guided 3D-aware Image Editingcited by 62
  3. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  4. 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51
  5. 2022StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objectscited by 49
  6. 2022IFOR: Iterative Flow Minimization for Robotic Object Rearrangementcited by 42
  7. 2023M2T2: Multi-Task Masked Transformer for Object-centric Pick and Placecited by 40
  8. 2023Compositional Diffusion-Based Continuous Constraint Solverscited by 39
  9. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  10. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Controlcited by 32
  11. 2023Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangementcited by 30
  12. 2023Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangementcited by 29
  13. 2024CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordancecited by 29
  14. 2023SG-Bot: Object Rearrangement via Coarse-to-Fine Robotic Imagination on Scene Graphscited by 24
  15. 2023Language-Conditioned Imitation Learning with Base Skill Priors under Unstructured Datacited by 20
  16. 2022Sequence-Based Plan Feasibility Prediction for Efficient Task and Motion Planningcited by 19
  17. 2023ReorientDiff: Diffusion Model based Reorientation for Object Manipulationcited by 19
  18. 2024Composable Part-Based Manipulationcited by 17
  19. 2023Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Modelscited by 16
  20. 2022Transformers are Adaptable Task Plannerscited by 15
  21. 2024CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissectioncited by 15
  22. 2024LLM-enhanced Scene Graph Learning for Household Rearrangementcited by 12
  23. 2024"Set It Up!": Functional Object Arrangement with Compositional Generative Modelscited by 9
  24. 2023NOIR: Neural Signal Operated Intelligent Robots for Everyday Activitiescited by 8
  25. 2023Structural Concept Learning via Graph Attention for Multi-Level Rearrangement Planningcited by 7
  26. 2022Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placementcited by 5
  27. 2023Out of Sight, Still in Mind: Reasoning and Planning about Unobserved Objects with Video Tracking Enabled Memory Modelscited by 5
  28. 2024Grasp, See, and Place: Efficient Unknown Object Rearrangement with Policy Structure Priorcited by 5
  29. 2023Blocks2World: Controlling Realistic Scenes with Editable Primitivescited by 4
  30. 2023SceneScore: Learning a Cost Function for Object Arrangementcited by 4
  31. 2023Spatial-Language Attention Policies for Efficient Robot Learningcited by 3
  32. 2023Interactive Visual Task Learning for Robotscited by 3
  33. 2024Learning Planning Abstractions from Languagecited by 3
  34. 2025Slot-Level Robotic Placement via Visual Imitation from Single Human Videocited by 2
  35. 2023"Tidy Up the Table": Grounding Common-sense Objective for Tabletop Object Rearrangementcited by 1
  36. 2025Tidiness Score-Guided Monte Carlo Tree Search for Visual Tabletop Rearrangementcited by 1
  37. 2026GIF: Agentic Generation of Interactive and Functional Object Compositions for Robot Learningcited by 1
  38. 2023Knolling Bot: Teaching Robots the Human Notion of Tidinesscited by 0

50 most cited of 57 All 57 (JSON)