Agentic Visual Generation: From Generative Models to Agentic Control
On the map: LLM agents and tool use → Multimodal and embodied agents
Abstract (arXiv)
Visual generation is evolving from generative models used through a single invocation into agentic control processes that can plan, select tools, inspect intermediate synthesized outputs, revise failures, and reuse prior experience. In most existing systems, the controller is an LLM or VLM, while visual generation models serve as tools or executors. However, existing work lacks a consistent criterion for determining when a generation system becomes agentic. Planning depth, tool use, multi-role collaboration, and reinforcement learning are often treated as evidence of agenticity, even though none of them necessarily determines which generation decisions the controller can make. We organize the field according to what the controller can directly control in the generation process. At L1 Conditioning Control, the controller prepares the input to a predetermined generator but does not control which visual operation is executed. At L2 Execution Control, it selects and invokes actual generation, editing, rendering, or other content-modifying operations. At L3 Outcome-Adaptive Control, it observes an intermediate outcome and uses that observation to change a subsequent operation within the current task. At L4 Experience-Adaptive Control, it retains experience from completed tasks and uses that experience to change decisions on future tasks. L0 Fixed Support separately denotes generators, editors, evaluators, reward models, benchmarks, and fixed pipelines without a deployed controller that makes generation-level decisions. These levels describe a progressively broader decision-making scope rather than model size, system complexity, output quality, tool or role count, or training method. Applying this framework across image, video, editing, 3D, world, slide, and user-interface generation reveals how controller capabilities have evolved and how their mechanisms are distributed across levels.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 379 of its 386 references that are in the map. First posted on arXiv 6 Sep 2026.
Builds on
379 of 386 references.
Closest ancestors · 131
Same direction, newest first
- 2026GenRouter: Unified Workflow Routing for Agentic Image Generation17 Aug
- 2026AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design13 Aug
- 2026AVA-Encoder: Towards Agent-Native Video Representation Learning12 Aug
- 2026Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherence12 Aug
- 2026iARCS: Iterative Agentic RL for Controllable 3D Scene Generation6 Aug
- 2026Domain-Grounded Candidate Selection for Agentic Image Editing: A Shadow Removal Case6 Aug
- 2026VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing6 Aug
- 2026WorldClaw: Agentic 3D Open-World Generation at Scale5 Aug
- 2026ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation5 Aug
- 2026SeaSlides: Semantic Abstraction Layer for Agentic Slide Generation4 Aug
- 2026PosterMELD: Multi-Agent Paper-to-Poster Generation for Controllable Design Diversity with Editable Print-Ready Outputs3 Aug
- 2026CADIR: A Cross-Backend Editable Intermediate Representation for Agentic CAD Generation1 Aug
38 more
- 2026DrawAI: Agentic Benchmark and Workflow for Making Raster Images Editable1 Aug
- 2026VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System29 Jul
- 2026CinemaTraj: Composing Atomic Camera Trajectories for 3D Scenes with LLM Agents29 Jul
- 2026GS-Agent: Creating 4D Physical Worlds With Generative Simulation23 Jul
- 2026Agentic Designer: Progressive Multi-Agent Collaboration for Structure-Aware Interior Layout Generation23 Jul
- 2026ETPDesigner: Multi-Agent Orchestration for Interactive Multimodal Electronic Theater Program22 Jul
- 2026FilmWorld: Agentic Novel-to-Film Generation through Dynamic Cinematic World Modeling21 Jul
- 2026PhysAgent: Reflective Agentic Physics Control for Physically Plausible Video Generation17 Jul
- 2026Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budget14 Jul
- 2026SymbOmni: Evolving Agentic Omni Models via Symbolic Concept Learning13 Jul
- 2026Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generation6 Jul
- 2026CoGen3D: An Agentic Human-AI Co-Design Pipeline for 3D Asset Generation for Virtual Reality4 Jul
- 2026SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation2 Jul
- 2026SAGE: Structured Agentic Graph Editing for Software Diagrams1 Jul
- 2026PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks30 Jun
- 2026ManimAgent: Self-Evolving Multimodal Agents for Visual Education29 Jun
- 2026NaLA: A 3D Native LLM Layout Agent for High-quality 3D Scene Generation28 Jun
- 2026Self-Evolving Agentic Image Restoration via Deliberate Planning and Intuitive Execution27 Jun
- 2026RS-Gen: A Multi-Stage Agentic Framework for Reasoning and Search-Augmented Image Generation22 Jun
- 2026One Image is All You Need: Agentic One-Shot Image Generation via Text-Based World Models for Long-Tail Spatial Perception18 Jun
- 2026Taming I2V models for Image HOI Editing: A Cognitive Benchmark and Agentic Self-Correcting Framework17 Jun
- 2026OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation16 Jun
- 2026MUSE: Agentic 3D Scene Authoring via Memory-Grounded Incremental Requirement Satisfaction12 Jun
- 2026InterleaveThinker: Reinforcing Agentic Interleaved Generation11 Jun
- 2026HDSL: A Hierarchical Domain-Specific Language for Structured 3D Indoor Scene Generation and Localized Editing with LLM Agents8 Jun
- 2026VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation6 Jun
- 2026IEA: Amateur-Friendly Conversational Image Editing Agent via Three Stages of Multitask Alignment6 Jun
- 2026MetaPoint: Unlocking Precise Spatial Control in Agentic Visual Generation3 Jun
- 2026ViMax: Agentic Video Generation2 Jun
- 2026OctoT2I: A Self-Evolving Agentic Text-to-Image Router1 Jun
- 20263DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code31 May
- 2026APE: Agentic Prompt Enhancer for Image Generation and Editing29 May
- 2026GenClaw: Code-Driven Agentic Image Generation28 May
- 2026DirectorBench: Diagnosing Long-Form Video Generation with Personalized Multi-Agent Evaluation28 May
- 2026MangaFlow: An End-to-End Agentic Framework for Controllable Story to Manga Generation27 May
- 2026CogPortrait: Fine-Grained Eye-Region Control in Portrait Animation via Hierarchical Agent Planning27 May
- 2026AgenticVBench: Can AI Agents Complete Real-World Post-Production Tasks?26 May
- 2026AtelierEval: Agentic Evaluation of Humans & LLMs as Text-to-Image Prompters21 May
50 newest of 131 All 131 (JSON)
Other references · 248
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
- 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
- 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
- 2022Video Diffusion Modelscited by 2,280
- 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
- 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
38 more
- 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
- 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
- 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
- 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
- 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
- 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
- 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
- 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
- 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
- 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
- 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
- 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
- 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
- 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 294
- 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
- 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
- 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
- 2022Optimizing Prompts for Text-to-Image Generationcited by 204
- 2023A Survey on Video Diffusion Modelscited by 203
- 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 182
- 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 174
- 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
- 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 146
- 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
- 2023Self-correcting LLM-controlled Diffusion Modelscited by 112
- 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
- 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 88
- 2024StableAnimator: High-Quality Identity-Preserving Human Image Animationcited by 85
- 2025PPTAgent: Generating and Evaluating Presentations Beyond Text-to-Slidescited by 83
- 2023Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Modelscited by 67
- 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 60
- 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 58
- 2023MotionEditor: Editing Video Motion via Content-Aware Diffusioncited by 56
- 2023Visual Programming for Text-to-Image Generation and Evaluationcited by 53
- 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Promptscited by 53
- 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 52
- 2025Captain Cinema: Towards Short Movie Generationcited by 48
50 most cited of 248 All 248 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026OmniHarness: Harnessing Generalizable Visual Generation via Symbolic Policy Learning
- 2026Autonomous Video Generation with Counterfactual Controllability for Self-Evolving World Models
- 2026VideoGen-Agent: Reinforcing Video Generation Agents
- 2026An Action Is Worth One Patch: Unified World-Action Modeling with PatchWAM
- 2025Much Ado About Noising: Dispelling the Myths of Generative Robotic Control
- 2025Simulating the Visual World with Artificial Intelligence: A Roadmap
- 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Review
- 2025Dreamland: Controllable World Creation with Simulator and Generative Models