GEMS: Agent-Native Multimodal Generation with Memory and Skills
Not on the map. Why?
Abstract (arXiv)
Recent multimodal generation models have achieved remarkable progress on general-purpose generation tasks, yet continue to struggle with complex instructions and specialized downstream tasks. Inspired by the success of advanced agent frameworks such as Claude Code, we propose GEMS (Agent-Native Multimodal GEneration with Memory and Skills), a framework that pushes beyond the inherent limitations of foundational models on both general and downstream tasks. GEMS is built upon three core components. Agent Loop introduces a structured multi-agent framework that iteratively improves generation quality through closed-loop optimization. Agent Memory provides a persistent, trajectory-level memory that hierarchically stores both factual states and compressed experiential summaries, enabling a global view of the optimization process while reducing redundancy. Agent Skill offers an extensible collection of domain-specific expertise with on-demand loading, allowing the system to effectively handle diverse downstream applications. Across five mainstream tasks and four downstream tasks, evaluated on multiple generative backends, GEMS consistently achieves significant performance gains. Most notably, it enables the lightweight 6B model Z-Image-Turbo to surpass the state-of-the-art Nano Banana 2 on GenEval2, demonstrating the effectiveness of agent harness in extending model capabilities beyond their original limits.
Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 70 of its 71 references that are in the map. First posted on arXiv 30 Mar 2026.
Builds on
70 of 71 references.
References · 70
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2020Denoising Diffusion Implicit Modelscited by 9,655
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
- 2022Scalable Diffusion Models with Transformerscited by 5,931
- 2022Classifier-Free Diffusion Guidancecited by 5,717
- 2022Flow Matching for Generative Modelingcited by 5,426
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
38 more
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
- 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
- 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
- 2025A-MEM: Agentic Memory for LLM Agentscited by 1,000
- 2025Qwen-Image Technical Reportcited by 973
- 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
- 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
- 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
- 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
- 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
- 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
- 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
- 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
- 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 289
- 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learningcited by 252
- 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 247
- 2025Seedream 4.0: Toward Next-generation Multimodal Image Generationcited by 233
- 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 225
- 2022Optimizing Prompts for Text-to-Image Generationcited by 204
- 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
- 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
- 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
- 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
- 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
- 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
- 2025OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generationcited by 81
- 2026XSkill: Continual Learning from Experience and Skills in Multimodal Agentscited by 71
- 2025From Reflection to Perfection: Scaling Inference-Time Optimization for Text-to-Image Diffusion Models via Reflection Tuningcited by 53
- 2026SkillNet: Create, Evaluate, and Connect AI Skillscited by 48
- 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflectioncited by 42
- 2025GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluationcited by 42
- 2025FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlightingcited by 39
50 most cited of 70 All 70 (JSON)
Built on by
Citing papers per year: 2026: 16.
16 papers.
Sort:
By area:
- 2026GenEvolve: Self-Evolving Image Generation Agents via Tool-Orchestrated Visual Experience Distillationcited by 15
- 2026Qwen-Image-Agent: Bridging the Context Gap in Real-World Image Generationcited by 8
- 2026SCOPE: Structured Decomposition and Conditional Skill Orchestration for Complex Image Generationcited by 6
- 2026Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skillscited by 6
- 2026Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modelingcited by 5
- 2026COMFYCLAW: Self-Evolving Skill Harnesses for Image Generation Workflowscited by 5
- 2026Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budgetcited by 5
- 2026GenRouter: Unified Workflow Routing for Agentic Image Generationcited by 4
- 2026Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMscited by 3
- 2026Ace-Skill: Bootstrapping Multimodal Agents with Prioritized and Clustered Evolutioncited by 2
- 2026Editable Visual Designcited by 0
- 2026WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editingcited by 0
4 more
- 2026Agentic Visual Generation: From Generative Models to Agentic Controlcited by 0
- 2026AutoRef: Harness Optimization for Agentic Multi-Reference Image Generationcited by 0
- 2026VIF-Bench: Evaluating Visual Instruction Following in Multi-Reference Image Generationcited by 0
- 2026MIRA: A Musical Intent Refinement Agent for Aligning Text-to-Music Generation with User Intentcited by 0
- 2026MIRA: A Musical Intent Refinement Agent for Aligning Text-to-Music Generation with User Intent7 Oct 2026
- 2026VIF-Bench: Evaluating Visual Instruction Following in Multi-Reference Image Generation29 Sep 2026
- 2026AutoRef: Harness Optimization for Agentic Multi-Reference Image Generation28 Sep 2026
- 2026Agentic Visual Generation: From Generative Models to Agentic Control6 Sep 2026
- 2026WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editing4 Sep 2026
- 2026Editable Visual Design3 Sep 2026
- 2026GenRouter: Unified Workflow Routing for Agentic Image Generation17 Aug 2026
- 2026Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills24 Jul 2026
- 2026Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budget14 Jul 2026
- 2026COMFYCLAW: Self-Evolving Skill Harnesses for Image Generation Workflows2 Jul 2026
- 2026Qwen-Image-Agent: Bridging the Context Gap in Real-World Image Generation25 Jun 2026
- 2026GenEvolve: Self-Evolving Image Generation Agents via Tool-Orchestrated Visual Experience Distillation20 May 2026
4 more
- 2026Ace-Skill: Bootstrapping Multimodal Agents with Prioritized and Clustered Evolution9 May 2026
- 2026SCOPE: Structured Decomposition and Conditional Skill Orchestration for Complex Image Generation8 May 2026
- 2026Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs1 May 2026
- 2026Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modeling30 Apr 2026
- 2026Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budgetcited by 5
- 2026GenRouter: Unified Workflow Routing for Agentic Image Generationcited by 4
- 2026WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editingcited by 0
- 2026Agentic Visual Generation: From Generative Models to Agentic Controlcited by 0
- 2026AutoRef: Harness Optimization for Agentic Multi-Reference Image Generationcited by 0
- 2026AutoRef: Harness Optimization for Agentic Multi-Reference Image Generation28 Sep 2026
- 2026Agentic Visual Generation: From Generative Models to Agentic Control6 Sep 2026
- 2026WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editing4 Sep 2026
- 2026GenRouter: Unified Workflow Routing for Agentic Image Generation17 Aug 2026
- 2026Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budget14 Jul 2026
- 2026GenEvolve: Self-Evolving Image Generation Agents via Tool-Orchestrated Visual Experience Distillationcited by 15
- 2026Qwen-Image-Agent: Bridging the Context Gap in Real-World Image Generationcited by 8
- 2026SCOPE: Structured Decomposition and Conditional Skill Orchestration for Complex Image Generationcited by 6
- 2026Ace-Skill: Bootstrapping Multimodal Agents with Prioritized and Clustered Evolutioncited by 2
- 2026Editable Visual Designcited by 0
- 2026VIF-Bench: Evaluating Visual Instruction Following in Multi-Reference Image Generationcited by 0
- 2026MIRA: A Musical Intent Refinement Agent for Aligning Text-to-Music Generation with User Intentcited by 0
- 2026MIRA: A Musical Intent Refinement Agent for Aligning Text-to-Music Generation with User Intent7 Oct 2026
- 2026VIF-Bench: Evaluating Visual Instruction Following in Multi-Reference Image Generation29 Sep 2026
- 2026Editable Visual Design3 Sep 2026
- 2026Qwen-Image-Agent: Bridging the Context Gap in Real-World Image Generation25 Jun 2026
- 2026GenEvolve: Self-Evolving Image Generation Agents via Tool-Orchestrated Visual Experience Distillation20 May 2026
- 2026Ace-Skill: Bootstrapping Multimodal Agents with Prioritized and Clustered Evolution9 May 2026
- 2026SCOPE: Structured Decomposition and Conditional Skill Orchestration for Complex Image Generation8 May 2026