mlsanityData to 14 Oct 2026

iARCS: Iterative Agentic RL for Controllable 3D Scene Generation

arXiv 2608.06161 ↗First posted 6 Aug 2026cs.AIcited by 1

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Synthetic 3D scene generation is increasingly used as a data source for computer vision and embodied AI, but existing generators often optimize perceptual realism without reliably satisfying task-critical functional constraints. This mismatch limits the usefulness of synthetic data for downstream training, where accessibility, traversability, and spatial rule compliance are often essential. We present iARCS, an iterative agentic reinforcement learning framework that adapts a pretrained scene generator to naturallanguage task requirements. iARCS uses a two-phase strategy: universal-reward pretraining to improve physical plausibility and layout quality, followed by task-specific finetuning with LLM-generated reward programs that are iteratively refined from training feedback. Experiments show improved constraint fidelity on walkability, reachability, and clearance-focused tasks, effective task-specific constraint optimization, and competitive scene diversity. We further show that data generated by iARCS improves a base generator, supporting its value as a practical synthetic data generation tool rather than only a controllable scene editing method.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 52 of its 71 references that are in the map. First posted on arXiv 6 Aug 2026.

Builds on

52 of 71 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026SAGE: Scalable Agentic 3D Scene Generation for Embodied AI10 Feb
  2. 2025SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent24 Sep

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 2022Scalable Diffusion Models with Transformerscited by 6,054
  5. 2022Classifier-Free Diffusion Guidancecited by 5,847
  6. 2022Flow Matching for Generative Modelingcited by 5,702
  7. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  8. 2014Going Deeper with Convolutionscited by 4,241
  9. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  10. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  11. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  12. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 604
38 more
  1. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  2. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  3. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 450
  4. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 380
  5. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  6. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
  7. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 311
  8. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 268
  9. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 234
  10. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 224
  11. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesiscited by 166
  12. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 160
  13. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 125
  14. 2024InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Priorcited by 117
  15. 2024GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splattingcited by 109
  16. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 96
  17. 2021Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphscited by 84
  18. 2024Large-scale Reinforcement Learning for Diffusion Modelscited by 78
  19. 2023LEGO-Net: Learning Regular Rearrangements of Objects in Roomscited by 76
  20. 2024DreamReward: Text-to-3D Generation with Human Preferencecited by 56
  21. 2023CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusioncited by 54
  22. 2025MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaversecited by 51
  23. 2024Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databasescited by 45
  24. 2015Text to 3D Scene Generation with Rich Lexical Groundingcited by 36
  25. 2024Mixed Diffusion for 3D Indoor Scene Synthesiscited by 32
  26. 2025Global-Local Tree Search in VLMs for 3D Indoor Scene Generationcited by 29
  27. 2025Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoningcited by 28
  28. 2025DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimizationcited by 27
  29. 2025FlairGPT: Repurposing LLMs for Interior Designscited by 24
  30. 2025SuperDec: 3D Scene Decomposition with Superquadric Primitivescited by 21
  31. 2025Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewardscited by 11
  32. 2025CHOrD: Generation of Collision-Free, House-Scale, and Organized Digital Twins for 3D Indoor Scenes with Controllable Floor Plans and Optimal Layoutscited by 10
  33. 2025Steerable Scene Generation with Post Training and Inference-Time Searchcited by 9
  34. 2025DreamCS: Geometry-Aware Text-to-3D Generation with Unpaired 3D Reward Supervisioncited by 7
  35. 2023Rearrange Indoor Scenes for Human-Robot Co-Activitycited by 5
  36. 20263D-Layout-R1: Structured Reasoning for Language-Instructed Spatial Editingcited by 4
  37. 2025Uncertainty-aware Reward Design Processcited by 1
  38. 2025MVRoom: Controllable 3D Indoor Scene Generation with Multi-View Diffusion Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Agentic Visual Generation: From Generative Models to Agentic Controlcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution
  2. 2025Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation
  3. 2026SceneReVis: A Self-Reflective Vision-Grounded Framework for 3D Indoor Scene Synthesis via Multi-Turn RL
  4. 2025Agentic 3D Scene Generation with Spatially Contextualized VLMs
  5. 2026AC3S: Adaptive Conditioning for 3D-Aware Synthetic Data Generation
  6. 2026MUSE: Agentic 3D Scene Authoring via Memory-Grounded Incremental Requirement Satisfaction
  7. 2026SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation
  8. 2026Scaling Sim-to-Real VLA Reinforcement Learning with Generative 3D Worlds