mlsanityData to 7 Oct 2026

Beyond Text Prompts: Visual-to-Visual Generation as A Unified Paradigm

arXiv 2605.12271 ↗First posted 12 May 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Humans often specify and create through visual artifacts: typography sheets, sketches, reference images, and annotated scenes. Yet modern visual generators still ask users to serialize this intent into text, a bottleneck that compresses signals like spatial structure, exact appearance, and glyph shape. We propose textbfvisual-to-visual (V2V) generation, in which the user conditions a generative model with a visual specification page rather than a text prompt. The page is not an edit target, but a visual document that specifies the desired output. We introduce V2V-Zero, a training-free framework that exposes this interface in existing vision-language model (VLM) conditioned generators by replacing text-only conditioning with final-layer hidden states extracted from visual pages, exploiting the fact that the frozen VLM already maps both text and images into the generator's conditioning space. On GenEval, V2V-Zero reaches 0.85 with a frozen Qwen-Image backbone, closely matching its optimized text-to-image performance without fine-tuning. To evaluate the broader V2V space, we introduce Simple-V2V Bench, spanning seven visual-conditioning tasks and seven models, including GPT Image 2, Nano Banana 2, Seedream 5.0 Lite, open-weight baselines, and a video extension. V2V-Zero scores 32.7/100, outperforming evaluated open-weight image baselines and revealing a clear capability hierarchy: attribute binding is strong, content generation is unreliable, and structural control remains hard even for commercial systems. A HunyuanVideo-1.5 extension scores 20.2/100, showing the interface transfers beyond images. Mechanistic analysis shows the default reasoning path is primarily visually routed, with 95.0% of conditioning-token attention mass on visual-page hidden states.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 57 of its 60 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

57 of 60 references.

References · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  10. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
38 more
  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  3. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  4. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  5. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  6. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  7. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  8. 2025Qwen-Image Technical Reportcited by 973
  9. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  10. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  11. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  12. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  13. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  14. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  15. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  16. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  17. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  18. 2025VACE: All-in-One Video Creation and Editingcited by 467
  19. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  20. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  21. 2024OmniGen: Unified Image Generationcited by 388
  22. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  23. 2022Visual Prompting via Image Inpaintingcited by 248
  24. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  25. 2023TextDiffuser: Diffusion Models as Text Painterscited by 223
  26. 2023AnyText: Multilingual Visual Text Generation And Editingcited by 165
  27. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  28. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 151
  29. 2025HunyuanVideo 1.5 Technical Reportcited by 148
  30. 2025Seedream 3.0 Technical Reportcited by 146
  31. 2025HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformercited by 131
  32. 2023GlyphControl: Glyph Conditional Control for Visual Text Generationcited by 127
  33. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
  34. 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
  35. 2025Lumina-Image 2.0: A Unified and Efficient Image Generative Frameworkcited by 93
  36. 2022Character-Aware Models Improve Visual Text Renderingcited by 90
  37. 2025UNIC: Unified In-Context Video Editingcited by 53
  38. 2025VisualCloze: A Universal Image Generation Framework via Visual In-Context Learningcited by 39

50 most cited of 57 All 57 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing is Believing: Aligning Prompt Rewriting with Visual Anchors for Text-to-Image Generation
  2. 2026Large Language Models are Universal Reasoners for Visual Generation
  3. 2026All-in-One Conditioning for Text-to-Image Synthesis
  4. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation
  5. 2025LumiGen: An LVLM-Enhanced Iterative Framework for Fine-Grained Text-to-Image Generation
  6. 2025Rethinking Prompt Design for Inference-time Scaling in Text-to-Visual Generation
  7. 2025ConceptMix++: Leveling the Playing Field in Text-to-Image Benchmarking via Iterative Prompt Optimization
  8. 2025Object-level Visual Prompts for Compositional Image Generation