mlsanityData to 7 Oct 2026

Spot-Compose: A Framework for Open-Vocabulary Object Retrieval and Drawer Manipulation in Point Clouds

arXiv 2404.12440 ↗First posted 18 Apr 2024cs.RO cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

In recent years, modern techniques in deep learning and large-scale datasets have led to impressive progress in 3D instance segmentation, grasp pose estimation, and robotics. This allows for accurate detection directly in 3D scenes, object- and environment-aware grasp prediction, as well as robust and repeatable robotic manipulation. This work aims to integrate these recent methods into a comprehensive framework for robotic interaction and manipulation in human-centric environments. Specifically, we leverage 3D reconstructions from a commodity 3D scanner for open-vocabulary instance segmentation, alongside grasp pose estimation, to demonstrate dynamic picking of objects, and opening of drawers. We show the performance and robustness of our model in two sets of real-world experiments including dynamic object retrieval and drawer opening, reporting a 51% and 82% success rate respectively. Code of our framework as well as videos are available on: https://spot-compose.github.io/.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 35 of its 49 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

35 of 49 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Mask R-CNNcited by 7,259
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,471
  6. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  7. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  8. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
  9. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  10. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  11. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  12. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
23 more
  1. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 237
  2. 2022AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domainscited by 205
  3. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  4. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 141
  5. 2017Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matchingcited by 124
  6. 2022Superpoint Transformer for 3D Scene Instance Segmentationcited by 115
  7. 2017Grasp Pose Detection in Point Cloudscited by 107
  8. 2023GOAT: GO to Any Thingcited by 89
  9. 2019Associatively Segmenting Instances and Semantics in Point Cloudscited by 86
  10. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Viewscited by 68
  11. 2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
  12. 2023Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labelscited by 64
  13. 2021Learning 3D Semantic Segmentation with only 2D Image Supervisioncited by 60
  14. 2019Antipodal Robotic Grasping using Generative Residual Convolutional Neural Networkcited by 59
  15. 2019Spherical Kernel for Efficient Graph Convolution on 3D Point Cloudscited by 57
  16. 2021End-to-end Trainable Deep Neural Network for Robotic Grasp Detection and Semantic Segmentation from RGBcited by 44
  17. 2023Open-vocabulary Panoptic Segmentation with Embedding Modulationcited by 39
  18. 20224D-StOP: Panoptic Segmentation of 4D LiDAR using Spatio-temporal Object Proposal Generation and Aggregationcited by 35
  19. 2024OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Roboticscited by 33
  20. 20223D Segmentation of Humans in Point Clouds with Synthetic Datacited by 24
  21. 2024ICGNet: A Unified Approach for Instance-Centric Graspingcited by 16
  22. 2019Robust and Adaptive Door Operation with a Mobile Robotcited by 13
  23. 2018The CoSTAR Block Stacking Dataset: Learning with Workspace Constraintscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 7; 2026: 1.

14 papers.

Sort:

By area:

  1. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 53
  2. 2024Search3D: Hierarchical Open-Vocabulary 3D Segmentationcited by 31
  3. 2024S2O: Static to Openable Enhancement for Articulated 3D Objectscited by 22
  4. 2024P2P-Bridge: Diffusion Bridges for 3D Point Cloud Denoisingcited by 18
  5. 2024Lost & Found: Tracking Changes from Egocentric Observations in 3D Dynamic Scene Graphscited by 10
  6. 2024ConceptAgent: LLM-Driven Precondition Grounding and Tree Search for Robust Task Planning and Executioncited by 8
  7. 2024OpenDAS: Open-Vocabulary Domain Adaptation for 2D and 3D Segmentationcited by 6
  8. 2025Efficient Point Clouds Upsampling via Flow Matchingcited by 4
  9. 2025SLAG: Scalable Language-Augmented Gaussian Splattingcited by 4
  10. 2025OpenCity3D: What do Vision-Language Models know about Urban Environments?cited by 3
  11. 2025Action-guided generation of 3D functionality segmentation datacited by 2
  12. 2025HouseLayout3D: A Benchmark and Training-Free Baseline for 3D Layout Estimation in the Wildcited by 1
2 more
  1. 2025PUFM++: Point Cloud Upsampling via Enhanced Flow Matchingcited by 0
  2. 2026Vernata: Self-Supervised Learning of LiDAR Point Representationscited by 0