# Generative Design through Quality-Diversity Data Synthesis and Language Models

arXiv: https://arxiv.org/abs/2405.09997 · HTML: https://mlsanity.com/paper/2405.09997/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 24 of its 53 references that are in the map.
- First posted on arXiv 16 May 2024.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- [SkexGen: Autoregressive Generation of CAD Construction Sequences with Disentangled Codebooks](https://mlsanity.com/paper/2207.04632/) (2022)
- [Quality and Diversity Optimization: A Unifying Modular Framework](https://mlsanity.com/paper/1708.09251/) (2017)
- [Hierarchical Neural Coding for Controllable CAD Model Generation](https://mlsanity.com/paper/2307.00149/) (2023)
- [SolidGen: An Autoregressive Model for Direct B-rep Synthesis](https://mlsanity.com/paper/2203.13944/) (2022)
- [Level Generation Through Large Language Models](https://mlsanity.com/paper/2302.05817/) (2023)
- [MarioGPT: Open-Ended Text2Level Generation through Large Language Models](https://mlsanity.com/paper/2302.05981/) (2023)
- [Data-Efficient Design Exploration through Surrogate-Assisted Illumination](https://mlsanity.com/paper/1806.05865/) (2018)
- [Unsupervised Behaviour Discovery with Quality-Diversity Optimisation](https://mlsanity.com/paper/2106.05648/) (2021)
- [Deep Surrogate Assisted MAP-Elites for Automated Hearthstone Deckbuilding](https://mlsanity.com/paper/2112.03534/) (2021)
- [Unsupervised Learning and Exploration of Reachable Outcome Space](https://mlsanity.com/paper/1909.05508/) (2019)
- [Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolution](https://mlsanity.com/paper/2104.13424/) (2021)
- [MAP-Elites with Descriptor-Conditioned Gradients and Archive Distillation into a Single Policy](https://mlsanity.com/paper/2303.03832/) (2023)
- [Empowering Quality Diversity in Dungeon Design with Interactive Constrained MAP-Elites](https://mlsanity.com/paper/1906.05175/) (2019)
- [Finding Game Levels with the Right Difficulty in a Few Trials through Intelligent Trial-and-Error](https://mlsanity.com/paper/2005.07677/) (2020)
- [Designing Air Flow with Surrogate-assisted Phenotypic Niching](https://mlsanity.com/paper/2105.04256/) (2021)
- [ARCH-Elites: Quality-Diversity for Urban Design](https://mlsanity.com/paper/2104.08774/) (2021)
- [Prototype Discovery using Quality-Diversity](https://mlsanity.com/paper/1807.09488/) (2018)
- [BOP-Elites, a Bayesian Optimisation Approach to Quality Diversity Search with Black-Box descriptor functions](https://mlsanity.com/paper/2307.09326/) (2023)

## Built on by (most cited first, 3 of 3)

- [Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Reset](https://mlsanity.com/paper/2409.17126/) (2024)
- [BikeBench: A Bicycle Design Benchmark for Generative Models with Objectives and Constraints](https://mlsanity.com/paper/2508.00830/) (2025)
- [CoMa: Contextual Massing Generation with Vision-Language Models](https://mlsanity.com/paper/2601.08464/) (2026)

## Abstract (arXiv)

Two fundamental challenges face generative models in engineering applications: the acquisition of high-performing, diverse datasets, and the adherence to precise constraints in generated designs. We propose a novel approach combining optimization, constraint satisfaction, and language models to tackle these challenges in architectural design. Our method uses Quality-Diversity (QD) to generate a diverse, high-performing dataset. We then fine-tune a language model with this dataset to generate high-level designs. These designs are then refined into detailed, constraint-compliant layouts using the Wave Function Collapse algorithm. Our system demonstrates reliable adherence to textual guidance, enabling the generation of layouts with targeted architectural and performance features. Crucially, our results indicate that data synthesized through the evolutionary search of QD not only improves overall model performance but is essential for the model's ability to closely adhere to textual guidance. This improvement underscores the pivotal role evolutionary computation can play in creating the datasets key to training generative models for design. Web article at https://tilegpt.github.io