# Prompt Evolution for Generative AI: A Classifier-Guided Approach

arXiv: https://arxiv.org/abs/2305.16347 · HTML: https://mlsanity.com/paper/2305.16347/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 17 of 17)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021)
- [Re-Imagen: Retrieval-Augmented Text-to-Image Generator](https://mlsanity.com/paper/2209.14491/) (2022)
- [Fine-tuning language models to find agreement among humans with diverse preferences](https://mlsanity.com/paper/2211.15006/) (2022)
- [TResNet: High Performance GPU-Dedicated Architecture](https://mlsanity.com/paper/2003.13630/) (2020)
- [ArtEmis: Affective Language for Visual Art](https://mlsanity.com/paper/2101.07396/) (2021)
- [Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models](https://mlsanity.com/paper/2207.13038/) (2022)
- [A Taxonomy of Prompt Modifiers for Text-To-Image Generation](https://mlsanity.com/paper/2204.13988/) (2022)
- [RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions](https://mlsanity.com/paper/2302.09466/) (2023)
- [Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts](https://mlsanity.com/paper/2109.08857/) (2021)
- [Evolutionary Multitasking for Multiobjective Continuous Optimization: Benchmark Problems, Performance Metrics and Baseline Results](https://mlsanity.com/paper/1706.02766/) (2017)
- [Grimm in Wonderland: Prompt Engineering with Midjourney to Illustrate Fairytales](https://mlsanity.com/paper/2302.08961/) (2023)

## Built on by (most cited first, 9 of 9)

- [Large Language and Text-to-3D Models for Engineering Design Optimization](https://mlsanity.com/paper/2307.01230/) (2023)
- [Enhancing Decision-Making in Optimization through LLM-Assisted Inference: A Neural Networks Perspective](https://mlsanity.com/paper/2405.07212/) (2024)
- [Generative AI-based Prompt Evolution Engineering Design Optimization With Vision-Language Model](https://mlsanity.com/paper/2406.09143/) (2024)
- [LLM2TEA: An Agentic AI Designer for Discovery with Generative Evolutionary Multitasking](https://mlsanity.com/paper/2406.14917/) (2024)
- [A Plug-and-Play Multi-Criteria Guidance for Diverse In-Betweening Human Motion Generation](https://mlsanity.com/paper/2508.01590/) (2025)
- [Automating Creativity](https://mlsanity.com/paper/2405.06915/) (2024)
- [Prompt Migration: Stabilizing GenAI Applications with Evolving Large Language Models](https://mlsanity.com/paper/2507.05573/) (2025)
- [Evolutionary Computation as Natural Generative AI](https://mlsanity.com/paper/2510.08590/) (2025)
- [Inference-Time Optimization of Prompt Embeddings in Diffusion Models: A Comparison of sep-CMA-ES and Adam](https://mlsanity.com/paper/2511.03913/) (2025)

## Abstract (arXiv)

Synthesis of digital artifacts conditioned on user prompts has become an important paradigm facilitating an explosion of use cases with generative AI. However, such models often fail to connect the generated outputs and desired target concepts/preferences implied by the prompts. Current research addressing this limitation has largely focused on enhancing the prompts before output generation or improving the model's performance up front. In contrast, this paper conceptualizes prompt evolution, imparting evolutionary selection pressure and variation during the generative process to produce multiple outputs that satisfy the target concepts/preferences better. We propose a multi-objective instantiation of this broader idea that uses a multi-label image classifier-guided approach. The predicted labels from the classifiers serve as multiple objectives to optimize, with the aim of producing diversified images that meet user preferences. A novelty of our evolutionary algorithm is that the pre-trained generative model gives us implicit mutation operations, leveraging the model's stochastic generative capability to automate the creation of Pareto-optimized images more faithful to user preferences.