mlsanityData to 7 Oct 2026

GEMS: Agent-Native Multimodal Generation with Memory and Skills

arXiv 2603.28088 ↗First posted 30 Mar 2026cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

Recent multimodal generation models have achieved remarkable progress on general-purpose generation tasks, yet continue to struggle with complex instructions and specialized downstream tasks. Inspired by the success of advanced agent frameworks such as Claude Code, we propose GEMS (Agent-Native Multimodal GEneration with Memory and Skills), a framework that pushes beyond the inherent limitations of foundational models on both general and downstream tasks. GEMS is built upon three core components. Agent Loop introduces a structured multi-agent framework that iteratively improves generation quality through closed-loop optimization. Agent Memory provides a persistent, trajectory-level memory that hierarchically stores both factual states and compressed experiential summaries, enabling a global view of the optimization process while reducing redundancy. Agent Skill offers an extensible collection of domain-specific expertise with on-demand loading, allowing the system to effectively handle diverse downstream applications. Across five mainstream tasks and four downstream tasks, evaluated on multiple generative backends, GEMS consistently achieves significant performance gains. Most notably, it enables the lightweight 6B model Z-Image-Turbo to surpass the state-of-the-art Nano Banana 2 on GenEval2, demonstrating the effectiveness of agent harness in extending model capabilities beyond their original limits.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 70 of its 71 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

70 of 71 references.

References · 70

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2022Flow Matching for Generative Modelingcited by 5,426
  11. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  12. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
38 more
  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  2. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  4. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  5. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  6. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  7. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  8. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,000
  9. 2025Qwen-Image Technical Reportcited by 973
  10. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  11. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  12. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  13. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  14. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
  15. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  16. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  17. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  18. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  19. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  20. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 289
  21. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learningcited by 252
  22. 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 247
  23. 2025Seedream 4.0: Toward Next-generation Multimodal Image Generationcited by 233
  24. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 225
  25. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  26. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
  27. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  28. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  29. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  30. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
  31. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  32. 2025OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generationcited by 81
  33. 2026XSkill: Continual Learning from Experience and Skills in Multimodal Agentscited by 71
  34. 2025From Reflection to Perfection: Scaling Inference-Time Optimization for Text-to-Image Diffusion Models via Reflection Tuningcited by 53
  35. 2026SkillNet: Create, Evaluate, and Connect AI Skillscited by 48
  36. 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflectioncited by 42
  37. 2025GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluationcited by 42
  38. 2025FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlightingcited by 39

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 16.

16 papers.

Sort:

By area:

  1. 2026GenEvolve: Self-Evolving Image Generation Agents via Tool-Orchestrated Visual Experience Distillationcited by 15
  2. 2026Qwen-Image-Agent: Bridging the Context Gap in Real-World Image Generationcited by 8
  3. 2026SCOPE: Structured Decomposition and Conditional Skill Orchestration for Complex Image Generationcited by 6
  4. 2026Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skillscited by 6
  5. 2026Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modelingcited by 5
  6. 2026COMFYCLAW: Self-Evolving Skill Harnesses for Image Generation Workflowscited by 5
  7. 2026Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budgetcited by 5
  8. 2026GenRouter: Unified Workflow Routing for Agentic Image Generationcited by 4
  9. 2026Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMscited by 3
  10. 2026Ace-Skill: Bootstrapping Multimodal Agents with Prioritized and Clustered Evolutioncited by 2
  11. 2026Editable Visual Designcited by 0
  12. 2026WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editingcited by 0
4 more
  1. 2026Agentic Visual Generation: From Generative Models to Agentic Controlcited by 0
  2. 2026AutoRef: Harness Optimization for Agentic Multi-Reference Image Generationcited by 0
  3. 2026VIF-Bench: Evaluating Visual Instruction Following in Multi-Reference Image Generationcited by 0
  4. 2026MIRA: A Musical Intent Refinement Agent for Aligning Text-to-Music Generation with User Intentcited by 0