# Generative AI-based Prompt Evolution Engineering Design Optimization With Vision-Language Model

arXiv: https://arxiv.org/abs/2406.09143 · HTML: https://mlsanity.com/paper/2406.09143/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [Shap-E: Generating Conditional 3D Implicit Functions](https://mlsanity.com/paper/2305.02463/) (2023)
- [SDM-NET: Deep Generative Network for Structured Deformable Mesh](https://mlsanity.com/paper/1908.04520/) (2019)
- [Drag-guided diffusion models for vehicle image generation](https://mlsanity.com/paper/2306.09935/) (2023)
- [Prompt Evolution for Generative AI: A Classifier-Guided Approach](https://mlsanity.com/paper/2305.16347/) (2023)
- [Large Language and Text-to-3D Models for Engineering Design Optimization](https://mlsanity.com/paper/2307.01230/) (2023)

## Built on by (most cited first, 3 of 3)

- [From Idea to CAD: A Language Model-Driven Multi-Agent System for Collaborative Design](https://mlsanity.com/paper/2503.04417/) (2025)
- [LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMs](https://mlsanity.com/paper/2506.11148/) (2025)
- [LLM2TEA: An Agentic AI Designer for Discovery with Generative Evolutionary Multitasking](https://mlsanity.com/paper/2406.14917/) (2024)

## Abstract (arXiv)

Engineering design optimization requires an efficient combination of a 3D shape representation, an optimization algorithm, and a design performance evaluation method, which is often computationally expensive. We present a prompt evolution design optimization (PEDO) framework contextualized in a vehicle design scenario that leverages a vision-language model for penalizing impractical car designs synthesized by a generative model. The backbone of our framework is an evolutionary strategy coupled with an optimization objective function that comprises a physics-based solver and a vision-language model for practical or functional guidance in the generated car designs. In the prompt evolutionary search, the optimizer iteratively generates a population of text prompts, which embed user specifications on the aerodynamic performance and visual preferences of the 3D car designs. Then, in addition to the computational fluid dynamics simulations, the pre-trained vision-language model is used to penalize impractical designs and, thus, foster the evolutionary algorithm to seek more viable designs. Our investigations on a car design optimization problem show a wide spread of potential car designs generated at the early phase of the search, which indicates a good diversity of designs in the initial populations, and an increase of over 20% in the probability of generating practical designs compared to a baseline framework without using a vision-language model. Visual inspection of the designs against the performance results demonstrates prompt evolution as a very promising paradigm for finding novel designs with good optimization performance while providing ease of use in specifying design specifications and preferences via a natural language interface.