# Enhance Image-to-Image Generation with LLaVA-generated Prompts

arXiv: https://arxiv.org/abs/2406.01956 · HTML: https://mlsanity.com/paper/2406.01956/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 69 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 30 of 42)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models](https://mlsanity.com/paper/2210.14896/) (2022)
- [Efficient Diffusion Training via Min-SNR Weighting Strategy](https://mlsanity.com/paper/2303.09556/) (2023)
- [Survival Prediction Across Diverse Cancer Types Using Neural Networks](https://mlsanity.com/paper/2404.08713/) (2024)
- [Can Language Understand Depth?](https://mlsanity.com/paper/2207.01077/) (2022)
- [Instruct-Imagen: Image Generation with Multi-modal Instruction](https://mlsanity.com/paper/2401.01952/) (2024)
- [Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networks](https://mlsanity.com/paper/2311.10944/) (2023)
- [Convolutional neural network classification of cancer cytopathology images: taking breast cancer as an example](https://mlsanity.com/paper/2404.08279/) (2024)
- [Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate](https://mlsanity.com/paper/2402.02769/) (2024)
- [Exploring Diverse Methods in Visual Question Answering](https://mlsanity.com/paper/2404.13565/) (2024)
- [Contextual Hourglass Network for Semantic Segmentation of High Resolution Aerial Imagery](https://mlsanity.com/paper/1810.12813/) (2018)
- [WorDepth: Variational Language Prior for Monocular Depth Estimation](https://mlsanity.com/paper/2404.03635/) (2024)
- [Regional Style and Color Transfer](https://mlsanity.com/paper/2404.13880/) (2024)
- [AutoReP: Automatic ReLU Replacement for Fast Private Network Inference](https://mlsanity.com/paper/2308.10134/) (2023)
- [Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models](https://mlsanity.com/paper/2403.17359/) (2024)
- [MaxK-GNN: Extremely Fast GPU Kernel Design for Accelerating Graph Neural Networks Training](https://mlsanity.com/paper/2312.08656/) (2023)
- [A Comparative Study on Enhancing Prediction in Social Network Advertisement through Data Augmentation](https://mlsanity.com/paper/2404.13812/) (2024)
- [Research on Splicing Image Detection Algorithms Based on Natural Image Statistical Characteristics](https://mlsanity.com/paper/2404.16296/) (2024)
- [Enhanced Detection Classification via Clustering SVM for Various Robot Collaboration Task](https://mlsanity.com/paper/2405.03026/) (2024)
- [Zero-shot Synthesis with Group-Supervised Learning](https://mlsanity.com/paper/2009.06586/) (2020)
- [Grow-and-Clip: Informative-yet-Concise Evidence Distillation for Answer Explanation](https://mlsanity.com/paper/2201.05088/) (2022)
- [Cross-Task Multi-Branch Vision Transformer for Facial Expression and Mask Wearing Classification](https://mlsanity.com/paper/2404.14606/) (2024)
- [Utilizing Deep Learning to Optimize Software Development Processes](https://mlsanity.com/paper/2404.13630/) (2024)
- [Adaptive speed planning for Unmanned Vehicle Based on Deep Reinforcement Learning](https://mlsanity.com/paper/2404.17379/) (2024)
- [T-GAE: Transferable Graph Autoencoder for Network Alignment](https://mlsanity.com/paper/2310.03272/) (2023)

## Built on by (most cited first, 20 of 20)

- [From Mind to Machine: The Rise of Manus AI as a Fully Autonomous Digital Agent](https://mlsanity.com/paper/2505.02024/) (2025)
- [Mitigating the Risk of Health Inequity Exacerbated by Large Language Models](https://mlsanity.com/paper/2410.05180/) (2024)
- [XMeCap: Meme Caption Generation with Sub-Image Adaptability](https://mlsanity.com/paper/2407.17152/) (2024)
- [Adaptive Fault Tolerance Mechanisms of Large Language Models in Cloud Computing Environments](https://mlsanity.com/paper/2503.12228/) (2025)
- [Research on Large Language Model Cross-Cloud Privacy Protection and Collaborative Training based on Federated Learning](https://mlsanity.com/paper/2503.12226/) (2025)
- [Cloud-Based AI Systems: Leveraging Large Language Models for Intelligent Fault Detection and Autonomous Self-Healing](https://mlsanity.com/paper/2505.11743/) (2025)
- [Data Augmentation Through Random Style Replacement](https://mlsanity.com/paper/2504.10563/) (2025)
- [Cross-Cloud Data Privacy Protection: Optimizing Collaborative Mechanisms of AI Systems by Integrating Federated Learning and LLMs](https://mlsanity.com/paper/2505.13292/) (2025)
- [Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training](https://mlsanity.com/paper/2409.04707/) (2024)
- [LLM-Driven E-Commerce Marketing Content Optimization: Balancing Creativity and Conversion](https://mlsanity.com/paper/2505.23809/) (2025)
- [Bias Evaluation and Mitigation in Retrieval-Augmented Medical Question-Answering Systems](https://mlsanity.com/paper/2503.15454/) (2025)
- [A Comprehensive Review of AI Agents: Transforming Possibilities in Technology and Beyond](https://mlsanity.com/paper/2508.11957/) (2025)
- [Curriculum Guided Reinforcement Learning for Efficient Multi Hop Retrieval Augmented Generation](https://mlsanity.com/paper/2505.17391/) (2025)
- [Instruction Tuning and CoT Prompting for Contextual Medical QA with LLMs](https://mlsanity.com/paper/2506.12182/) (2025)
- [Transforming Movie Recommendations with Advanced Machine Learning: A Study of NMF, SVD,and K-Means Clustering](https://mlsanity.com/paper/2407.08916/) (2024)
- [Enhancing Exchange Rate Forecasting with Explainable Deep Learning Models](https://mlsanity.com/paper/2410.19241/) (2024)
- [DeepRAG: Integrating Hierarchical Reasoning and Process Supervision for Biomedical Multi-Hop QA](https://mlsanity.com/paper/2506.00671/) (2025)
- [Pre-trained Graphformer-based Ranking at Web-scale Search (Extended Abstract)](https://mlsanity.com/paper/2409.16590/) (2024)
- [Generative Pre-trained Ranking Model with Over-parameterization at Web-Scale (Extended Abstract)](https://mlsanity.com/paper/2409.16594/) (2024)
- [Leveraging Vision-Language Large Models for Interpretable Video Action Recognition with Semantic Tokenization](https://mlsanity.com/paper/2509.05695/) (2025)

## Abstract (arXiv)

This paper presents a novel approach to enhance image-to-image generation by leveraging the multimodal capabilities of the Large Language and Vision Assistant (LLaVA). We propose a framework where LLaVA analyzes input images and generates textual descriptions, hereinafter LLaVA-generated prompts. These prompts, along with the original image, are fed into the image-to-image generation pipeline. This enriched representation guides the generation process towards outputs that exhibit a stronger resemblance to the input image. Extensive experiments demonstrate the effectiveness of LLaVA-generated prompts in promoting image similarity. We observe a significant improvement in the visual coherence between the generated and input images compared to traditional methods. Future work will explore fine-tuning LLaVA prompts for increased control over the creative process. By providing more specific details within the prompts, we aim to achieve a delicate balance between faithfulness to the original image and artistic expression in the generated outputs.