mlsanityData to 7 Oct 2026

Prompt Refinement with Image Pivot for Text-to-Image Generation

arXiv 2407.00247 ↗First posted 28 Jun 2024cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

For text-to-image generation, automatically refining user-provided natural language prompts into the keyword-enriched prompts favored by systems is essential for the user experience. Such a prompt refinement process is analogous to translating the prompt from "user languages" into "system languages". However, the scarcity of such parallel corpora makes it difficult to train a prompt refinement model. Inspired by zero-shot machine translation techniques, we introduce Prompt Refinement with Image Pivot (PRIP). PRIP innovatively uses the latent representation of a user-preferred image as an intermediary "pivot" between the user and system languages. It decomposes the refinement process into two data-rich tasks: inferring representations of user-preferred images from user languages and subsequently translating image representations into system languages. Thus, it can leverage abundant data for training. Extensive experiments show that PRIP substantially outperforms a wide range of baselines and effectively transfers to unseen systems in a zero-shot manner.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 25 of its 46 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

25 of 46 references.

References · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2021Variational Diffusion Modelscited by 1,145
  9. 2019How Can We Know What Language Models Know?cited by 953
  10. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  11. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  12. 2018Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Surveycited by 459
13 more
  1. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  2. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  3. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  4. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  5. 2021Neural Machine Translation for Low-Resource Languages: A Surveycited by 108
  6. 2019Neural Machine Translation: A Review and Surveycited by 99
  7. 2021BERTese: Learning to Speak to BERTcited by 82
  8. 2022Investigating Prompt Engineering in Diffusion Modelscited by 58
  9. 2023PromptMagician: Interactive Prompt Engineering for Text-to-Image Creationcited by 58
  10. 2023Control3D: Towards Controllable Text-to-3D Generationcited by 46
  11. 2024TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Modelscited by 30
  12. 2023A Prompt Log Analysis of Text-to-Image Generation Systemscited by 24
  13. 2023Manipulating Embeddings of Stable Diffusion Promptscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 3.

11 papers.

  1. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  2. 2024Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLMcited by 22
  3. 2025The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generationcited by 21
  4. 2024IPAdapter-Instruct: Resolving Ambiguity in Image-based Conditioning using Instruct Promptscited by 18
  5. 2025RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scalingcited by 9
  6. 2026Show, Don't Tell: Morphing Latent Reasoning into Image Generationcited by 7
  7. 2026SCMAPR: Self-Correcting Multi-Agent Prompt Refinement for Complex-Scenario Text-to-Video Generationcited by 4
  8. 2026GenRouter: Unified Workflow Routing for Agentic Image Generationcited by 4
  9. 2025Foundations of GenIRcited by 1
  10. 2025InstructEngine: Instruction-driven Text-to-Image Alignmentcited by 1
  11. 2025TextTIGER: Text-based Intelligent Generation with Entity Prompt Refinement for Text-to-Image Generationcited by 1