mlsanityData to 7 Oct 2026

Agentic Visual Generation: From Generative Models to Agentic Control

arXiv 2609.06758 ↗First posted 6 Sep 2026cs.CVcited by 0

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Visual generation is evolving from generative models used through a single invocation into agentic control processes that can plan, select tools, inspect intermediate synthesized outputs, revise failures, and reuse prior experience. In most existing systems, the controller is an LLM or VLM, while visual generation models serve as tools or executors. However, existing work lacks a consistent criterion for determining when a generation system becomes agentic. Planning depth, tool use, multi-role collaboration, and reinforcement learning are often treated as evidence of agenticity, even though none of them necessarily determines which generation decisions the controller can make. We organize the field according to what the controller can directly control in the generation process. At L1 Conditioning Control, the controller prepares the input to a predetermined generator but does not control which visual operation is executed. At L2 Execution Control, it selects and invokes actual generation, editing, rendering, or other content-modifying operations. At L3 Outcome-Adaptive Control, it observes an intermediate outcome and uses that observation to change a subsequent operation within the current task. At L4 Experience-Adaptive Control, it retains experience from completed tasks and uses that experience to change decisions on future tasks. L0 Fixed Support separately denotes generators, editors, evaluators, reward models, benchmarks, and fixed pipelines without a deployed controller that makes generation-level decisions. These levels describe a progressively broader decision-making scope rather than model size, system complexity, output quality, tool or role count, or training method. Applying this framework across image, video, editing, 3D, world, slide, and user-interface generation reveals how controller capabilities have evolved and how their mechanisms are distributed across levels.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 379 of its 386 references that are in the map. First posted on arXiv 6 Sep 2026.

Builds on

379 of 386 references.

Closest ancestors · 131

Same direction, newest first

  1. 2026GenRouter: Unified Workflow Routing for Agentic Image Generation17 Aug
  2. 2026AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design13 Aug
  3. 2026AVA-Encoder: Towards Agent-Native Video Representation Learning12 Aug
  4. 2026Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherence12 Aug
  5. 2026iARCS: Iterative Agentic RL for Controllable 3D Scene Generation6 Aug
  6. 2026Domain-Grounded Candidate Selection for Agentic Image Editing: A Shadow Removal Case6 Aug
  7. 2026VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing6 Aug
  8. 2026WorldClaw: Agentic 3D Open-World Generation at Scale5 Aug
  9. 2026ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation5 Aug
  10. 2026SeaSlides: Semantic Abstraction Layer for Agentic Slide Generation4 Aug
  11. 2026PosterMELD: Multi-Agent Paper-to-Poster Generation for Controllable Design Diversity with Editable Print-Ready Outputs3 Aug
  12. 2026CADIR: A Cross-Backend Editable Intermediate Representation for Agentic CAD Generation1 Aug
38 more
  1. 2026DrawAI: Agentic Benchmark and Workflow for Making Raster Images Editable1 Aug
  2. 2026VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System29 Jul
  3. 2026CinemaTraj: Composing Atomic Camera Trajectories for 3D Scenes with LLM Agents29 Jul
  4. 2026GS-Agent: Creating 4D Physical Worlds With Generative Simulation23 Jul
  5. 2026Agentic Designer: Progressive Multi-Agent Collaboration for Structure-Aware Interior Layout Generation23 Jul
  6. 2026ETPDesigner: Multi-Agent Orchestration for Interactive Multimodal Electronic Theater Program22 Jul
  7. 2026FilmWorld: Agentic Novel-to-Film Generation through Dynamic Cinematic World Modeling21 Jul
  8. 2026PhysAgent: Reflective Agentic Physics Control for Physically Plausible Video Generation17 Jul
  9. 2026Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budget14 Jul
  10. 2026SymbOmni: Evolving Agentic Omni Models via Symbolic Concept Learning13 Jul
  11. 2026Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generation6 Jul
  12. 2026CoGen3D: An Agentic Human-AI Co-Design Pipeline for 3D Asset Generation for Virtual Reality4 Jul
  13. 2026SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation2 Jul
  14. 2026SAGE: Structured Agentic Graph Editing for Software Diagrams1 Jul
  15. 2026PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks30 Jun
  16. 2026ManimAgent: Self-Evolving Multimodal Agents for Visual Education29 Jun
  17. 2026NaLA: A 3D Native LLM Layout Agent for High-quality 3D Scene Generation28 Jun
  18. 2026Self-Evolving Agentic Image Restoration via Deliberate Planning and Intuitive Execution27 Jun
  19. 2026RS-Gen: A Multi-Stage Agentic Framework for Reasoning and Search-Augmented Image Generation22 Jun
  20. 2026One Image is All You Need: Agentic One-Shot Image Generation via Text-Based World Models for Long-Tail Spatial Perception18 Jun
  21. 2026Taming I2V models for Image HOI Editing: A Cognitive Benchmark and Agentic Self-Correcting Framework17 Jun
  22. 2026OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation16 Jun
  23. 2026MUSE: Agentic 3D Scene Authoring via Memory-Grounded Incremental Requirement Satisfaction12 Jun
  24. 2026InterleaveThinker: Reinforcing Agentic Interleaved Generation11 Jun
  25. 2026HDSL: A Hierarchical Domain-Specific Language for Structured 3D Indoor Scene Generation and Localized Editing with LLM Agents8 Jun
  26. 2026VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation6 Jun
  27. 2026IEA: Amateur-Friendly Conversational Image Editing Agent via Three Stages of Multitask Alignment6 Jun
  28. 2026MetaPoint: Unlocking Precise Spatial Control in Agentic Visual Generation3 Jun
  29. 2026ViMax: Agentic Video Generation2 Jun
  30. 2026OctoT2I: A Self-Evolving Agentic Text-to-Image Router1 Jun
  31. 20263DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code31 May
  32. 2026APE: Agentic Prompt Enhancer for Image Generation and Editing29 May
  33. 2026GenClaw: Code-Driven Agentic Image Generation28 May
  34. 2026DirectorBench: Diagnosing Long-Form Video Generation with Personalized Multi-Agent Evaluation28 May
  35. 2026MangaFlow: An End-to-End Agentic Framework for Controllable Story to Manga Generation27 May
  36. 2026CogPortrait: Fine-Grained Eye-Region Control in Portrait Animation via Hierarchical Agent Planning27 May
  37. 2026AgenticVBench: Can AI Agents Complete Real-World Post-Production Tasks?26 May
  38. 2026AtelierEval: Agentic Evaluation of Humans & LLMs as Text-to-Image Prompters21 May

50 newest of 131 All 131 (JSON)

Other references · 248

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  9. 2022Video Diffusion Modelscited by 2,280
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  11. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  12. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
38 more
  1. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  2. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  3. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  5. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  6. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  7. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  8. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  9. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  10. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  11. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  12. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  13. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  14. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  15. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 294
  16. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  17. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  18. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
  19. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  20. 2023A Survey on Video Diffusion Modelscited by 203
  21. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 182
  22. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 174
  23. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  24. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 146
  25. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  26. 2023Self-correcting LLM-controlled Diffusion Modelscited by 112
  27. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
  28. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 88
  29. 2024StableAnimator: High-Quality Identity-Preserving Human Image Animationcited by 85
  30. 2025PPTAgent: Generating and Evaluating Presentations Beyond Text-to-Slidescited by 83
  31. 2023Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Modelscited by 67
  32. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 60
  33. 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 58
  34. 2023MotionEditor: Editing Video Motion via Content-Aware Diffusioncited by 56
  35. 2023Visual Programming for Text-to-Image Generation and Evaluationcited by 53
  36. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Promptscited by 53
  37. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 52
  38. 2025Captain Cinema: Towards Short Movie Generationcited by 48

50 most cited of 248 All 248 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026OmniHarness: Harnessing Generalizable Visual Generation via Symbolic Policy Learning
  2. 2026Autonomous Video Generation with Counterfactual Controllability for Self-Evolving World Models
  3. 2026VideoGen-Agent: Reinforcing Video Generation Agents
  4. 2026An Action Is Worth One Patch: Unified World-Action Modeling with PatchWAM
  5. 2025Much Ado About Noising: Dispelling the Myths of Generative Robotic Control
  6. 2025Simulating the Visual World with Artificial Intelligence: A Roadmap
  7. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Review
  8. 2025Dreamland: Controllable World Creation with Simulator and Generative Models