mlsanityData to 14 Oct 2026

PlaceIt3D: Language-Guided Object Placement in Real 3D Scenes

arXiv 2505.05288 ↗First posted 8 May 2025cs.CV cs.AI cs.ROcited by 9

Not on the map. Why?

Abstract (arXiv)

We introduce the novel task of Language-Guided Object Placement in Real 3D Scenes. Our model is given a 3D scene's point cloud, a 3D asset, and a textual prompt broadly describing where the 3D asset should be placed. The task here is to find a valid placement for the 3D asset that respects the prompt. Compared with other language-guided localization tasks in 3D scenes such as grounding, this task has specific challenges: it is ambiguous because it has multiple valid solutions, and it requires reasoning about 3D geometric relationships and free space. We inaugurate this task by proposing a new benchmark and evaluation protocol. We also introduce a new dataset for training 3D LLMs on this task, as well as the first method to serve as a non-trivial baseline. We believe that this challenging task and our new benchmark could become part of the suite of benchmarks used to evaluate and compare generalist 3D LLM models.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 48 of its 63 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

48 of 63 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  7. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  8. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  9. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  10. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  11. 2017Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentationscited by 620
  12. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
36 more
  1. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  2. 2017Large-scale Point Cloud Semantic Segmentation with Superpoint Graphscited by 385
  3. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
  4. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  5. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  6. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 268
  7. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 251
  8. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  9. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  10. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  11. 2021TEACh: Task-driven Embodied Agents that Chatcited by 197
  12. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  13. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesiscited by 166
  14. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 152
  15. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 147
  16. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  17. 2022Superpoint Transformer for 3D Scene Instance Segmentationcited by 116
  18. 2024SAMPart3D: Segment Any Part in 3D Objectscited by 97
  19. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 96
  20. 2023AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homescited by 82
  21. 2023LEGO-Net: Learning Regular Rearrangements of Objects in Roomscited by 76
  22. 2019SceneGraphNet: Neural Message Passing for 3D Indoor Scene Augmentationcited by 55
  23. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  24. 2024SceneCraft: Layout-Guided 3D Scene Generationcited by 41
  25. 2015Text to 3D Scene Generation with Rich Lexical Groundingcited by 36
  26. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  27. 2022ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenescited by 32
  28. 2021OPA: Object Placement Assessment Datasetcited by 28
  29. 2024Reason3D: Searching and Reasoning 3D Segmentation via Large Language Modelcited by 28
  30. 2025FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placementcited by 25
  31. 2025Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3Dcited by 23
  32. 2017SceneSeer: 3D Scene Design with Natural Languagecited by 21
  33. 2024VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigationcited by 21
  34. 2023TopNet: Transformer-based Object Placement Network for Image Compositingcited by 16
  35. 2024Seeing the Unseen: Visual Common Sense for Semantic Placementcited by 9
  36. 2024OCTO+: A Suite for Automatic Open-Vocabulary Object Placement in Mixed Realitycited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangementcited by 7
  2. 20263D-Layout-R1: Structured Reasoning for Language-Instructed Spatial Editingcited by 4
  3. 2026Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modelingcited by 2
  4. 2026SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenescited by 1
  5. 2026Beyond Single Object: Learning 3D Relations with Large Language Modelscited by 1
  6. 2026HiddenObjects: Scalable Diffusion-Distilled Spatial Priors for Object Placementcited by 0
  7. 2026ReRef-3D: A Benchmark for Spatial Referring Expression-Guided 3D Scene Rearrangementcited by 0
  8. 2026SPHERE: Adaptive VR Indoor Scene Generation via LLM-Enhanced Spatial Preference Learning and Human-in-the-Loop RLcited by 0
  9. 2026Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editingcited by 0