mlsanityData to 14 Oct 2026

SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model

arXiv 2403.13064 ↗First posted 19 Mar 2024cs.CVcited by 96

Not on the map. Why?

Abstract (arXiv)

We introduce SceneScript, a method that directly produces full scene models as a sequence of structured language commands using an autoregressive, token-based approach. Our proposed scene representation is inspired by recent successes in transformers & LLMs, and departs from more traditional methods which commonly describe scenes as meshes, voxel grids, point clouds or radiance fields. Our method infers the set of structured language commands directly from encoded visual data using a scene language encoder-decoder architecture. To train SceneScript, we generate and release a large-scale synthetic dataset called Aria Synthetic Environments consisting of 100k high-quality in-door scenes, with photorealistic and ground-truth annotated renders of egocentric scene walkthroughs. Our method gives state-of-the art results in architectural layout estimation, and competitive results in 3D object detection. Lastly, we explore an advantage for SceneScript, which is the ability to readily adapt to new commands via simple additions to the structured language, which we illustrate for tasks such as coarse 3D object part reconstruction.

Key facts. Cited by 96 papers in the map (data through 14 Oct 2026). Builds on 35 of its 54 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

35 of 54 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  5. 2015Fast R-CNNcited by 2,933
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  8. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  9. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  10. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 423
  11. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 366
  12. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 349
23 more
  1. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  2. 2021DeepCAD: A Deep Generative Network for Computer-Aided Design Modelscited by 271
  3. 2020PolyGen: An Autoregressive Generative Model of 3D Meshescited by 269
  4. 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 208
  5. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 200
  6. 2016Learning Shape Abstractions by Assembling Volumetric Primitivescited by 181
  7. 2022Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wildcited by 160
  8. 2020DeepSVG: A Hierarchical Generative Network for Vector Graphics Animationcited by 159
  9. 2021ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detectioncited by 157
  10. 2022SkexGen: Autoregressive Generation of CAD Construction Sequences with Disentangled Codebookscited by 125
  11. 2018PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Imagecited by 105
  12. 2018LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Imagecited by 95
  13. 2021Computer-Aided Design as Languagecited by 82
  14. 2022TorchSparse: Efficient Point Cloud Inference Enginecited by 73
  15. 2021SketchGen: Generating Constrained CAD Sketchescited by 68
  16. 2022Connecting the Dots: Floorplan Reconstruction Using Two-Level Queriescited by 49
  17. 2020SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scanscited by 44
  18. 2021Unsupervised Learning for Cuboid Shape Abstraction via Joint Segmentation from Point Cloudscited by 44
  19. 2017RoomNet: End-to-End Room Layout Estimationcited by 38
  20. 2019Floor-SP: Inverse CAD for Floorplans by Sequential Room-wise Shortest Pathcited by 36
  21. 2020PLAD: Learning to Infer Shape Programs with Pseudo-Labels and Approximate Distributionscited by 19
  22. 2022RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformerscited by 17
  23. 2022Point2Seq: Detecting 3D Objects as Sequencescited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 32; 2026: 54.

96 papers.

Sort:

By area:

  1. 2025MapAnything: Universal Feed-Forward Metric 3D Reconstructioncited by 314
  2. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  3. 2024MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Secondscited by 141
  4. 2025Sonata: Self-Supervised Learning of Reliable Point Representationscited by 110
  5. 2024SLAM3R: Real-Time Dense Scene Reconstruction from Monocular RGB Videoscited by 102
  6. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 96
  7. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 65
  8. 2025Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representationscited by 45
  9. 2024Autoregressive Models in Vision: A Surveycited by 39
  10. 2025Any4D: Unified Feed-Forward Metric 4D Reconstructioncited by 35
  11. 2025DINO in the Room: Leveraging 2D Foundation Models for 3D Segmentationcited by 31
  12. 2025N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Modelscited by 29
38 more
  1. 2024SPA: 3D Spatial-Awareness Enables Effective Embodied Representationcited by 28
  2. 2026ShapeR: Robust Conditional 3D Shape Generation from Casual Capturescited by 22
  3. 2025LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scanscited by 21
  4. 2025PanSt3R: Multi-view Consistent Panoptic Segmentationcited by 20
  5. 2025InternScenes: A Large-scale Simulatable Indoor Scene Dataset with Realistic Layoutscited by 20
  6. 2026VGG-T3: Offline Feed-Forward 3D Reconstruction at Scalecited by 18
  7. 2025ViSTA-SLAM: Visual SLAM with Symmetric Two-view Associationcited by 16
  8. 2025SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splattingcited by 15
  9. 20263DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Codecited by 15
  10. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 14
  11. 2026LoMa: Local Feature Matching Revisitedcited by 11
  12. 2025Ov3R: Open-Vocabulary Semantic 3D Reconstruction from RGB Videoscited by 10
  13. 2025'Eclair -- Extracting Content and Layout with Integrated Reading Order for Documentscited by 9
  14. 2025EgoM2P: Egocentric Multimodal Multitask Pretrainingcited by 9
  15. 2026LuxRemix: Lighting Decomposition and Remixing for Indoor Scenescited by 9
  16. 2024Reconstructing Animals and the Wildcited by 7
  17. 2024DI-PCG: Diffusion-based Efficient Inverse Procedural Content Generation for High-quality 3D Asset Creationcited by 7
  18. 2026Thinking with Spatial Code for Physical-World Video Reasoningcited by 7
  19. 2026Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3Dcited by 7
  20. 2026Vision as Unified Multimodal Generationcited by 7
  21. 2025ArcPro: Architectural Programs for Structured 3D Abstraction of Sparse Pointscited by 6
  22. 2025ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editingcited by 6
  23. 2026Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligencecited by 6
  24. 2026D'ej`a View: Looping Transformers for Multi-View 3D Reconstructioncited by 6
  25. 2026Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representationscited by 5
  26. 2026GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generationcited by 5
  27. 2026BrickNet: Graph-Backed Generative Brick Assemblycited by 5
  28. 2025Human-in-the-Loop Local Corrections of 3D Scene Layouts via Infillingcited by 4
  29. 2025Aligning Text, Images, and 3D Structure Token-by-Tokencited by 4
  30. 20263AM: 3egment Anything with Geometric Consistency in Videoscited by 4
  31. 2026TrajVG: 3D Trajectory-Coupled Visual Geometry Learningcited by 4
  32. 2026Mage: Multi-Axis Evaluation of LLM-Generated Executable Game Scenes Beyond Compile-Pass Ratecited by 4
  33. 2025Unposed Sparse Views Room Layout Reconstruction in the Age of Pretrain Modelcited by 3
  34. 2025Explaining Human Preferences via Metrics for Structured 3D Reconstructioncited by 3
  35. 2026SceneLinker: Compositional 3D Scene Generation via Semantic Scene Graph from RGB Sequencescited by 3
  36. 2026NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Datacited by 3
  37. 2026SuperFlex: Deformable Superquadrics for Point Cloud Decompositioncited by 3
  38. 2026X-Lens: Real-Time Metric Depth Estimation with Heterogeneous Camerascited by 3

50 most cited of 96 All 96 (JSON)