mlsanityData to 14 Oct 2026

ReRef-3D: A Benchmark for Spatial Referring Expression-Guided 3D Scene Rearrangement

arXiv 2608.16011 ↗First posted 17 Aug 2026cs.CL cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce ReRef-3D, a benchmark for language-guided placement in 3D scenes. It contains 33,826 instructions across 998 CLEVR-derived scenes, spanning 16 placement families and direct, one-hop, and two-hop references. Each instruction must be resolved into a valid new placement position. Given that an instruction defines a region of acceptable placements rather than one coordinate, our evaluation inserts a prediction into the scene, recomputes relations, and tests relation satisfaction and physical validity. Each instruction also includes a verified naturalized rewrite. After fine-tuning, LLaVA-3D, 3D-LLM, and PlaceIt3D produce valid placements for 68.3%, 31.6%, and 22.4% of instructions, respectively. Across models, relation satisfaction surpasses physical validity, relations such as nearest and between are the most difficult, and phrasing has minimal effect on performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 71 references that are in the map. First posted on arXiv 17 Aug 2026.

Builds on

33 of 71 references.

References · 33

  1. 2016Modeling Context in Referring Expressionscited by 1,063
  2. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  3. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  4. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  5. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  6. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  7. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  8. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  9. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  10. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  11. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  12. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
21 more
  1. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 251
  2. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  3. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
  4. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 136
  5. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 122
  6. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  7. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 99
  8. 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 77
  9. 2019CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressionscited by 73
  10. 2022StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Textscited by 68
  11. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 59
  12. 2022StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objectscited by 50
  13. 2022ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenescited by 32
  14. 2022Quality Controlled Paraphrase Generationcited by 31
  15. 2024Learning Visual Grounding from Generative Vision and Language Modelcited by 27
  16. 2025SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Datacited by 21
  17. 2023LGMCTS: Language-Guided Monte-Carlo Tree Search for Executable Semantic Object Rearrangementcited by 11
  18. 2025PlaceIt3D: Language-Guided Object Placement in Real 3D Scenescited by 9
  19. 2024LINGO-Space: Language-Conditioned Incremental Grounding for Spacecited by 8
  20. 2024ParaFusion: A Large-Scale LLM-Driven English Paraphrase Dataset Infused with High-Quality Lexical and Syntactic Diversitycited by 8
  21. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generationcited by 8

Built on by

Not cited yet.