Beyond Text Prompts: Visual-to-Visual Generation as A Unified Paradigm
Not on the map. Why?
Abstract (arXiv)
Humans often specify and create through visual artifacts: typography sheets, sketches, reference images, and annotated scenes. Yet modern visual generators still ask users to serialize this intent into text, a bottleneck that compresses signals like spatial structure, exact appearance, and glyph shape. We propose textbfvisual-to-visual (V2V) generation, in which the user conditions a generative model with a visual specification page rather than a text prompt. The page is not an edit target, but a visual document that specifies the desired output. We introduce V2V-Zero, a training-free framework that exposes this interface in existing vision-language model (VLM) conditioned generators by replacing text-only conditioning with final-layer hidden states extracted from visual pages, exploiting the fact that the frozen VLM already maps both text and images into the generator's conditioning space. On GenEval, V2V-Zero reaches 0.85 with a frozen Qwen-Image backbone, closely matching its optimized text-to-image performance without fine-tuning. To evaluate the broader V2V space, we introduce Simple-V2V Bench, spanning seven visual-conditioning tasks and seven models, including GPT Image 2, Nano Banana 2, Seedream 5.0 Lite, open-weight baselines, and a video extension. V2V-Zero scores 32.7/100, outperforming evaluated open-weight image baselines and revealing a clear capability hierarchy: attribute binding is strong, content generation is unreliable, and structural control remains hard even for commercial systems. A HunyuanVideo-1.5 extension scores 20.2/100, showing the interface transfers beyond images. Mechanistic analysis shows the default reasoning path is primarily visually routed, with 95.0% of conditioning-token attention mass on visual-page hidden states.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 57 of its 60 references that are in the map. First posted on arXiv 12 May 2026.
Builds on
57 of 60 references.
References · 57
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
- 2022Scalable Diffusion Models with Transformerscited by 5,931
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
- 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
38 more
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
- 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
- 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
- 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
- 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
- 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
- 2025Qwen-Image Technical Reportcited by 973
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
- 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
- 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
- 2024Emu3: Next-Token Prediction is All You Needcited by 761
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
- 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
- 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
- 2025VACE: All-in-One Video Creation and Editingcited by 467
- 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
- 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
- 2024OmniGen: Unified Image Generationcited by 388
- 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
- 2022Visual Prompting via Image Inpaintingcited by 248
- 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
- 2023TextDiffuser: Diffusion Models as Text Painterscited by 223
- 2023AnyText: Multilingual Visual Text Generation And Editingcited by 165
- 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
- 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 151
- 2025HunyuanVideo 1.5 Technical Reportcited by 148
- 2025Seedream 3.0 Technical Reportcited by 146
- 2025HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformercited by 131
- 2023GlyphControl: Glyph Conditional Control for Visual Text Generationcited by 127
- 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
- 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
- 2025Lumina-Image 2.0: A Unified and Efficient Image Generative Frameworkcited by 93
- 2022Character-Aware Models Improve Visual Text Renderingcited by 90
- 2025UNIC: Unified In-Context Video Editingcited by 53
- 2025VisualCloze: A Universal Image Generation Framework via Visual In-Context Learningcited by 39
50 most cited of 57 All 57 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Seeing is Believing: Aligning Prompt Rewriting with Visual Anchors for Text-to-Image Generation
- 2026Large Language Models are Universal Reasoners for Visual Generation
- 2026All-in-One Conditioning for Text-to-Image Synthesis
- 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation
- 2025LumiGen: An LVLM-Enhanced Iterative Framework for Fine-Grained Text-to-Image Generation
- 2025Rethinking Prompt Design for Inference-time Scaling in Text-to-Visual Generation
- 2025ConceptMix++: Leveling the Playing Field in Text-to-Image Benchmarking via Iterative Prompt Optimization
- 2025Object-level Visual Prompts for Compositional Image Generation