mlsanityData to 7 Oct 2026

Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models

arXiv 2312.04533 ↗First posted 7 Dec 2023cs.RO cs.CV cs.LGcited by 16

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

We introduce Dream2Real, a robotics framework which integrates vision-language models (VLMs) trained on 2D data into a 3D object rearrangement pipeline. This is achieved by the robot autonomously constructing a 3D representation of the scene, where objects can be rearranged virtually and an image of the resulting arrangement rendered. These renders are evaluated by a VLM, so that the arrangement which best satisfies the user instruction is selected and recreated in the real world with pick-and-place. This enables language-conditioned rearrangement to be performed zero-shot, without needing to collect a training dataset of example arrangements. Results on a series of real-world tasks show that this framework is robust to distractors, controllable by language, capable of understanding complex multi-object relations, and readily applicable to both tabletop and 6-DoF rearrangement tasks.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 62 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

53 of 62 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  2. 2023Open-World Object Manipulation using Pre-trained Vision-Language Models2 Mar
  3. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models23 Jul

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Segment Anythingcited by 8,158
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  7. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  8. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  9. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  10. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  11. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  12. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
38 more
  1. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  2. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  3. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  4. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  5. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 305
  6. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  7. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  8. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  9. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  10. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  11. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  12. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  13. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  14. 2022SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusioncited by 125
  15. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 105
  16. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 103
  17. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  18. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  19. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learningcited by 70
  20. 2023vMAP: Vectorised Object Mapping for Neural Field SLAMcited by 67
  21. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  22. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 57
  23. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  24. 2022StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objectscited by 49
  25. 2022LaTeRF: Label and Text Driven Object Radiance Fieldscited by 38
  26. 2022TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priorscited by 30
  27. 2023Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangementcited by 30
  28. 2023Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangementcited by 29
  29. 2021My House, My Rules: Learning Tidying Preferences with Graph Neural Networkscited by 27
  30. 2022MIRA: Mental Imagery for Robotic Affordancescited by 26
  31. 2023SG-Bot: Object Rearrangement via Coarse-to-Fine Robotic Imagination on Scene Graphscited by 24
  32. 2021Predicting Stable Configurations for Semantic Placement of Novel Objectscited by 23
  33. 2021Efficient and Interpretable Robot Manipulation with Graph Neural Networkscited by 21
  34. 2023CabiNet: Scaling Neural Collision Detection for Object Rearrangement with Procedural Scene Generationcited by 17
  35. 2022Transformers are Adaptable Task Plannerscited by 15
  36. 2022ReorientBot: Learning Object Reorientation for Specific-Posed Placementcited by 10
  37. 2023ConSOR: A Context-Aware Semantic Object Rearrangement Framework for Partially Arranged Scenescited by 6
  38. 2023SceneScore: Learning a Cost Function for Object Arrangementcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 4.

16 papers.

Sort:

By area:

  1. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  2. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  3. 2025Beyond the Destination: A Novel Benchmark for Exploration-Aware Embodied Question Answeringcited by 29
  4. 2024KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Modelscited by 22
  5. 2025EmbodiedSplat: Personalized Real-to-Sim-to-Real Navigation with Gaussian Splats from a Mobile Devicecited by 11
  6. 2026Text-Guided 6D Object Pose Rearrangement via Closed-Loop VLM Agentscited by 3
  7. 2025Hierarchical Language Models for Semantic Navigation and Manipulation in an Aerial-Ground Robotic Systemcited by 2
  8. 2025Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)cited by 2
  9. 2024NeRF-Aug: Data Augmentation for Robotics with Neural Radiance Fieldscited by 1
  10. 2025World Knowledge from AI Image Generation for Robot Controlcited by 1
  11. 2025Improving Generalization of Language-Conditioned Robot Manipulationcited by 1
  12. 2026Memory-Guided View Refinement for Dynamic Human-in-the-loop EQAcited by 1
4 more
  1. 2024Stimulating Imagination: Towards General-purpose "Something Something Placement"cited by 0
  2. 2024PACA: Perspective-Aware Cross-Attention Representation for Zero-Shot Scene Rearrangementcited by 0
  3. 2026presto: Efficient, Training-free, and Open-world Object Placement via Imaginary Searchcited by 0
  4. 2026Dynamic Manipulation with World-Action Models via Counterfactual Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Semantically Grounded Object Matching for Robust Robotic Scene Rearrangement
  2. 2025Learn from the Past: Language-conditioned Object Rearrangement with Large Language Models
  3. 2024HumanVLA: Towards Vision-Language Directed Object Rearrangement by Physical Humanoid
  4. 2025VULCAN: Tool-Augmented Multi Agents for Iterative 3D Object Arrangement
  5. 2026Grounding Vision and Language to 3D Masks for Long-Horizon Box Rearrangement
  6. 2025Goal-VLA: Image-Generative VLMs as Object-Centric World Models Empowering Zero-shot Robot Manipulation
  7. 2026Copy-Trasform-Paste: Zero-Shot Object-Object Alignment Guided by Vision-Language and Geometric Constraints
  8. 2026SAM3D-Guided Object-Centric Representation Alignment for Vision-Language-Action Models