mlsanityData to 7 Oct 2026

Enhance Image-to-Image Generation with LLaVA-generated Prompts

arXiv 2406.01956 ↗First posted 4 Jun 2024cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

This paper presents a novel approach to enhance image-to-image generation by leveraging the multimodal capabilities of the Large Language and Vision Assistant (LLaVA). We propose a framework where LLaVA analyzes input images and generates textual descriptions, hereinafter LLaVA-generated prompts. These prompts, along with the original image, are fed into the image-to-image generation pipeline. This enriched representation guides the generation process towards outputs that exhibit a stronger resemblance to the input image. Extensive experiments demonstrate the effectiveness of LLaVA-generated prompts in promoting image similarity. We observe a significant improvement in the visual coherence between the generated and input images compared to traditional methods. Future work will explore fine-tuning LLaVA prompts for increased control over the creative process. By providing more specific details within the prompts, we aim to achieve a delicate balance between faithfulness to the original image and artistic expression in the generated outputs.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 42 of its 69 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

42 of 69 references.

References · 42

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2024Large Language Models: A Surveycited by 506
  7. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  8. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 242
  9. 2024Survival Prediction Across Diverse Cancer Types Using Neural Networkscited by 80
  10. 2022Can Language Understand Depth?cited by 66
  11. 2024Instruct-Imagen: Image Generation with Multi-modal Instructioncited by 65
  12. 2023Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networkscited by 54
30 more
  1. 2024Convolutional neural network classification of cancer cytopathology images: taking breast cancer as an examplecited by 50
  2. 2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 46
  3. 2024Exploring Diverse Methods in Visual Question Answeringcited by 45
  4. 2018Contextual Hourglass Network for Semantic Segmentation of High Resolution Aerial Imagerycited by 42
  5. 2024WorDepth: Variational Language Prior for Monocular Depth Estimationcited by 34
  6. 2024Regional Style and Color Transfercited by 30
  7. 2023AutoReP: Automatic ReLU Replacement for Fast Private Network Inferencecited by 29
  8. 2024Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Modelscited by 27
  9. 2023MaxK-GNN: Extremely Fast GPU Kernel Design for Accelerating Graph Neural Networks Trainingcited by 25
  10. 2024A Comparative Study on Enhancing Prediction in Social Network Advertisement through Data Augmentationcited by 20
  11. 2024Research on Splicing Image Detection Algorithms Based on Natural Image Statistical Characteristicscited by 20
  12. 2020Zero-shot Synthesis with Group-Supervised Learningcited by 17
  13. 2024Enhanced Detection Classification via Clustering SVM for Various Robot Collaboration Taskcited by 17
  14. 2022Grow-and-Clip: Informative-yet-Concise Evidence Distillation for Answer Explanationcited by 16
  15. 2024Cross-Task Multi-Branch Vision Transformer for Facial Expression and Mask Wearing Classificationcited by 16
  16. 2024Utilizing Deep Learning to Optimize Software Development Processescited by 15
  17. 2024Adaptive speed planning for Unmanned Vehicle Based on Deep Reinforcement Learningcited by 15
  18. 2023T-GAE: Transferable Graph Autoencoder for Network Alignmentcited by 14
  19. 2024Few-shot Name Entity Recognition on StackOverflowcited by 14
  20. 2024Localization Through Particle Filter Powered Neural Network Estimated Monocular Camera Posescited by 11
  21. 2024Mix of Experts Language Model for Named Entity Recognitioncited by 11
  22. 2024Research on Image Recognition Technology Based on Multimodal Deep Learningcited by 11
  23. 2024AD-Aligning: Emulating Human-like Generalization for Cognitive Domain Adaptation in Deep Learningcited by 8
  24. 2020Deep Learning for Content-based Personalized Viewport Prediction of 360-Degree VR Videoscited by 7
  25. 2024A Stochastic GDA Method With Backtracking For Solving Nonconvex Concave Minimax Problemscited by 6
  26. 2024Research on Detection of Floating Objects in River and Lake Based on AI Intelligent Image Recognitioncited by 6
  27. 2021Encouraging Disentangled and Convex Representation with Controllable Interpolation Regularizationcited by 4
  28. 2024CoRMF: Criticality-Ordered Recurrent Mean Field Ising Solvercited by 3
  29. 2018Using an Ancillary Neural Network to Capture Weekends and Holidays in an Adjoint Neural Network Architecture for Intelligent Building Managementcited by 2
  30. 2021Performance analysis of facial recognition: A critical review through glass factorcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 7; 2025: 13.

20 papers.

Sort:

By area:

  1. 2025From Mind to Machine: The Rise of Manus AI as a Fully Autonomous Digital Agentcited by 33
  2. 2024Mitigating the Risk of Health Inequity Exacerbated by Large Language Modelscited by 21
  3. 2024XMeCap: Meme Caption Generation with Sub-Image Adaptabilitycited by 14
  4. 2025Adaptive Fault Tolerance Mechanisms of Large Language Models in Cloud Computing Environmentscited by 12
  5. 2025Research on Large Language Model Cross-Cloud Privacy Protection and Collaborative Training based on Federated Learningcited by 11
  6. 2025Cloud-Based AI Systems: Leveraging Large Language Models for Intelligent Fault Detection and Autonomous Self-Healingcited by 10
  7. 2025Data Augmentation Through Random Style Replacementcited by 9
  8. 2025Cross-Cloud Data Privacy Protection: Optimizing Collaborative Mechanisms of AI Systems by Integrating Federated Learning and LLMscited by 9
  9. 2024Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Trainingcited by 7
  10. 2025LLM-Driven E-Commerce Marketing Content Optimization: Balancing Creativity and Conversioncited by 7
  11. 2025Bias Evaluation and Mitigation in Retrieval-Augmented Medical Question-Answering Systemscited by 6
  12. 2025A Comprehensive Review of AI Agents: Transforming Possibilities in Technology and Beyondcited by 6
8 more
  1. 2025Curriculum Guided Reinforcement Learning for Efficient Multi Hop Retrieval Augmented Generationcited by 5
  2. 2025Instruction Tuning and CoT Prompting for Contextual Medical QA with LLMscited by 5
  3. 2024Transforming Movie Recommendations with Advanced Machine Learning: A Study of NMF, SVD,and K-Means Clusteringcited by 2
  4. 2024Enhancing Exchange Rate Forecasting with Explainable Deep Learning Modelscited by 2
  5. 2025DeepRAG: Integrating Hierarchical Reasoning and Process Supervision for Biomedical Multi-Hop QAcited by 2
  6. 2024Pre-trained Graphformer-based Ranking at Web-scale Search (Extended Abstract)cited by 0
  7. 2024Generative Pre-trained Ranking Model with Over-parameterization at Web-Scale (Extended Abstract)cited by 0
  8. 2025Leveraging Vision-Language Large Models for Interpretable Video Action Recognition with Semantic Tokenizationcited by 0