# NeuroPrompts: An Adaptive Framework to Optimize Prompts for Text-to-Image Generation

arXiv: https://arxiv.org/abs/2311.12229 · HTML: https://mlsanity.com/paper/2311.12229/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 21 of its 25 references that are in the map.
- First posted on arXiv 20 Nov 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models](https://mlsanity.com/paper/2210.14896/) (2022)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [Stable Bias: Analyzing Societal Representations in Diffusion Models](https://mlsanity.com/paper/2303.11408/) (2023)
- [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021)
- [NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics](https://mlsanity.com/paper/2112.08726/) (2021)
- [NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints](https://mlsanity.com/paper/2010.12884/) (2020)
- [NeuroCounterfactuals: Beyond Minimal-Edit Counterfactuals for Richer Data Augmentation](https://mlsanity.com/paper/2210.12365/) (2022)
- [I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation](https://mlsanity.com/paper/2212.09246/) (2022)
- [NeuroComparatives: Neuro-Symbolic Distillation of Comparative Knowledge](https://mlsanity.com/paper/2305.04978/) (2023)

## Built on by (most cited first, 10 of 10)

- [Enhancing Intent Understanding for Ambiguous prompt: A Human-Machine Co-Adaption Strategy](https://mlsanity.com/paper/2501.15167/) (2025)
- [PromptRL: Prompt Matters in RL for Flow-Based Image Generation](https://mlsanity.com/paper/2602.01382/) (2026)
- [VisualPrompter: Semantic-Aware Prompt Optimization with Visual Feedback for Text-to-Image Synthesis](https://mlsanity.com/paper/2506.23138/) (2025)
- [p-less Sampling: A Robust Hyperparameter-Free Approach for LLM Decoding](https://mlsanity.com/paper/2509.23234/) (2025)
- [A Systematic Post-Train Framework for Video Generation](https://mlsanity.com/paper/2604.25427/) (2026)
- [From Noise to Nuance: Advances in Deep Generative Image Models](https://mlsanity.com/paper/2412.09656/) (2024)
- [One Rewrite to Fix Them All? Type-Aware Repair Allocation for Text-to-Image Prompt Optimization](https://mlsanity.com/paper/2607.18724/) (2026)
- [AdaptaGen: Domain-Specific Image Generation through Hierarchical Semantic Optimization Framework](https://mlsanity.com/paper/2507.05621/) (2025)
- [EvoGen-Harness: Learning Where and How to Evolve Image-Generation Harnesses](https://mlsanity.com/paper/2610.00383/) (2026)
- [DOGS: Design-Space Sampling for Prompt-Driven Logo Generation](https://mlsanity.com/paper/2610.10760/) (2026)

## Abstract (arXiv)

Despite impressive recent advances in text-to-image diffusion models, obtaining high-quality images often requires prompt engineering by humans who have developed expertise in using them. In this work, we present NeuroPrompts, an adaptive framework that automatically enhances a user's prompt to improve the quality of generations produced by text-to-image models. Our framework utilizes constrained text decoding with a pre-trained language model that has been adapted to generate prompts similar to those produced by human prompt engineers. This approach enables higher-quality text-to-image generations and provides user control over stylistic features via constraint set specification. We demonstrate the utility of our framework by creating an interactive application for prompt enhancement and image generation using Stable Diffusion. Additionally, we conduct experiments utilizing a large dataset of human-engineered prompts for text-to-image generation and show that our approach automatically produces enhanced prompts that result in superior image quality. We make our code and a screencast video demo of NeuroPrompts publicly available.