mlsanityData to 7 Oct 2026

Grasp, See, and Place: Efficient Unknown Object Rearrangement with Policy Structure Prior

arXiv 2402.15402 ↗First posted 23 Feb 2024cs.RO cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We focus on the task of unknown object rearrangement, where a robot is supposed to re-configure the objects into a desired goal configuration specified by an RGB-D image. Recent works explore unknown object rearrangement systems by incorporating learning-based perception modules. However, they are sensitive to perception error, and pay less attention to task-level performance. In this paper, we aim to develop an effective system for unknown object rearrangement amidst perception noise. We theoretically reveal that the noisy perception impacts grasp and place in a decoupled way, and show such a decoupled structure is valuable to improve task optimality. We propose GSP, a dual-loop system with the decoupled structure as prior. For the inner loop, we learn a see policy for self-confident in-hand object matching. For the outer loop, we learn a grasp policy aware of object matching and grasp capability guided by task-level rewards. We leverage the foundation model CLIP for object matching, policy learning and self-termination. A series of experiments indicate that GSP can conduct unknown object rearrangement with higher completion rates and fewer steps.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 68 of its 100 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

68 of 100 references.

References · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  5. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  6. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  10. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  11. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  12. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
38 more
  1. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  2. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  3. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  4. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  5. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  6. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  7. 2017Embodied Question Answeringcited by 335
  8. 2022Masked Visual Pre-training for Motor Controlcited by 246
  9. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  10. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  11. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  12. 2019Soft Actor-Critic for Discrete Action Settingscited by 161
  13. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  14. 2017Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matchingcited by 124
  15. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 101
  16. 2020Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphscited by 83
  17. 2016Interactive Perception: Leveraging Action in Perception and Perception in Actioncited by 77
  18. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  19. 2019Online Replanning in Belief Space for Partially Observable Task and Motion Problemscited by 60
  20. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 57
  21. 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51
  22. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  23. 2022StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objectscited by 49
  24. 2020Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Imagecited by 45
  25. 2022IFOR: Iterative Flow Minimization for Robotic Object Rearrangementcited by 42
  26. 2020Learning 3D Dynamic Scene Representations for Robot Manipulationcited by 39
  27. 2021NeRP: Neural Rearrangement Planning for Unknown Objectscited by 33
  28. 2023A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Cluttercited by 33
  29. 2019Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Cluttercited by 30
  30. 2018Reinforcement Learning of Active Vision for Manipulating Objects under Occlusionscited by 25
  31. 2021OCRTOC: A Cloud-Based Competition and Benchmark for Robotic Grasping and Manipulationcited by 24
  32. 2019Monte-Carlo Tree Search for Efficient Visually Guided Rearrangement Planningcited by 23
  33. 2021Predicting Stable Configurations for Semantic Placement of Novel Objectscited by 23
  34. 2021Semantically Grounded Object Matching for Robust Robotic Scene Rearrangementcited by 21
  35. 2023Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulationcited by 21
  36. 2019Object Finding in Cluttered Scenes Using Interactive Perceptioncited by 18
  37. 2018Multi-View Picking: Next-best-view Reaching for Improved Grasping in Cluttercited by 15
  38. 2021How to select and use tools? : Active Perception of Target Objects Using Multimodal Deep Learningcited by 15

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 5.

5 papers.

  1. 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policycited by 17
  2. 2025ExploreVLM: Closed-Loop Robot Exploration Task Planning with Vision-Language Modelscited by 7
  3. 2025Efficient Alignment of Unconditioned Action Prior for Language-conditioned Pick and Place in Cluttercited by 3
  4. 2025GPA-RAM: Grasp-Pretraining Augmented Robotic Attention Mamba for Spatial Task Learningcited by 1
  5. 2025Dynamic Buffers: Cost-Efficient Planning for Tabletop Rearrangement with Stackingcited by 0