mlsanityData to 7 Oct 2026

SceneActBench: Can Agents Act on the 3D Scenes They See?

arXiv 2607.22393 ↗First posted 24 Jul 2026cs.AI cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Vision-language model (VLM) agents increasingly use tools to act on 3D scenes rather than only describe them. Existing 3D benchmarks score textual responses or single-object operations, leaving agent action on complete multi-object 3D scenes under evaluated. We present SceneActBench, a benchmark for visually conditioned action across five 3D tasks under a unified agent-environment loop. Given PNG images or sampled video frames and, where applicable, supplied 3D assets, an agent acts on a 3D environment. We evaluate each final output against hidden ground truth with task-specific geometric metrics. SceneActBench comprises five tasks built from 210 source instances, yielding 520 task cases including paired input conditions. Every task runs through one fixed agent loop to keep the comparison fair. Across eleven proprietary VLM configurations, Overall scores span 38.6-50.2, and none performs consistently well across tasks. We further analyse where and how failures manifest.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 41 references that are in the map. First posted on arXiv 24 Jul 2026.

Builds on

31 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents13 Feb
  2. 2025Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Models12 Feb
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan

Other references · 28

  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  2. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  3. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  4. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  5. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  6. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  7. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  8. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  9. 2023FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objectscited by 411
  10. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  11. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  12. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
16 more
  1. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  2. 2019What Do Single-view 3D Reconstruction Networks Learn?cited by 259
  3. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  4. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 195
  5. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  6. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesiscited by 159
  7. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  8. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 125
  9. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
  10. 2026Seed2.0 Model Card: Towards Intelligence Frontier for Real-World Complexitycited by 48
  11. 2025CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generationcited by 39
  12. 2025BlenderGym: Benchmarking Foundational Model Systems for Graphics Editingcited by 27
  13. 2024S2O: Static to Openable Enhancement for Articulated 3D Objectscited by 22
  14. 2026GameDevBench: Evaluating Agentic Capabilities Through Game Developmentcited by 21
  15. 20263DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Codecited by 9
  16. 2025M3DLayout: A Multi-Source Dataset of 3D Indoor Layouts and Structured Descriptions for 3D Generationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026BVB: Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blendercited by 1
  2. 20263DHarnessBench: Probing Agentic 3D-to-Code Capabilities of Frontier Vision-Language Modelscited by 0
  3. 2026Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering
  2. 2026SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes
  3. 2024Agent3D-Zero: An Agent for Zero-shot 3D Understanding
  4. 2025VisualActBench: Can VLMs See and Act like a Human?
  5. 2026SceneTeract: Probing and Improving Agent-Aware Activity Reasoning in 3D Indoor Scenes
  6. 2026SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation
  7. 2026Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena
  8. 2025VULCAN: Tool-Augmented Multi Agents for Iterative 3D Object Arrangement