# Generative Pre-Trained Transformer for Design Concept Generation: An Exploration

arXiv: https://arxiv.org/abs/2111.08489 · HTML: https://mlsanity.com/paper/2111.08489/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 61 references that are in the map.
- First posted on arXiv 16 Nov 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [Deep Generative Models in Engineering Design: A Review](https://mlsanity.com/paper/2110.10863/) (2021)
- [Few-shot Natural Language Generation for Task-Oriented Dialog](https://mlsanity.com/paper/2002.12328/) (2020)
- [TopologyGAN: Topology Optimization Using Generative Adversarial Networks Based on Physical Fields Over the Initial Domain](https://mlsanity.com/paper/2003.04685/) (2020)
- [Paraphrase Generation with Deep Reinforcement Learning](https://mlsanity.com/paper/1711.00279/) (2017)
- [Patent Claim Generation by Fine-Tuning OpenAI GPT-2](https://mlsanity.com/paper/1907.02052/) (2019)
- [Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet](https://mlsanity.com/paper/2102.08036/) (2021)
- [Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation](https://mlsanity.com/paper/1805.08191/) (2018)

## Built on by (most cited first, 17 of 17)

- [Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions](https://mlsanity.com/paper/2305.10435/) (2023)
- [Fluid Transformers and Creative Analogies: Exploring Large Language Models' Capacity for Augmenting Cross-Domain Analogical Creativity](https://mlsanity.com/paper/2302.12832/) (2023)
- [Natural Language Processing in-and-for Design Research](https://mlsanity.com/paper/2111.13827/) (2021)
- [Biologically Inspired Design Concept Generation Using Generative Pre-Trained Transformers](https://mlsanity.com/paper/2212.13196/) (2022)
- [Conceptual Design Generation Using Large Language Models](https://mlsanity.com/paper/2306.01779/) (2023)
- [Mapping the Design Space of Interactions in Human-AI Text Co-creation Tasks](https://mlsanity.com/paper/2303.06430/) (2023)
- [Opportunities for Large Language Models and Discourse in Engineering Design](https://mlsanity.com/paper/2306.09169/) (2023)
- [Reviewing Intelligent Cinematography: AI research for camera-based video production](https://mlsanity.com/paper/2405.05039/) (2024)
- [Large Language Model for Patent Concept Generation](https://mlsanity.com/paper/2409.00092/) (2024)
- [Elucidating STEM Concepts through Generative AI: A Multi-modal Exploration of Analogical Reasoning](https://mlsanity.com/paper/2308.10454/) (2023)
- [The Innovation Paradox: Concept Space Expansion with Diminishing Originality and the Promise of Creative AI](https://mlsanity.com/paper/2303.13300/) (2023)
- [ChatGPT Alternative Solutions: Large Language Models Survey](https://mlsanity.com/paper/2403.14469/) (2024)
- [Human-in-the-Loop: Quantitative Evaluation of 3D Models Generation by Large Language Models](https://mlsanity.com/paper/2509.07010/) (2025)
- [Generative Pre-Trained Transformers for Biologically Inspired Design](https://mlsanity.com/paper/2204.09714/) (2022)
- [LLM-Based Design Pattern Detection](https://mlsanity.com/paper/2502.18458/) (2025)
- [RECALL-MM: A Multimodal Dataset of Consumer Product Recalls for Risk Analysis using Computational Methods and Large Language Models](https://mlsanity.com/paper/2503.23213/) (2025)
- [Design-OS: A Specification-Driven Framework for Engineering System Design with a Control-Systems Design Case](https://mlsanity.com/paper/2603.20151/) (2026)

## Abstract (arXiv)

Novel concepts are essential for design innovation and can be generated with the aid of data stimuli and computers. However, current generative design algorithms focus on diagrammatic or spatial concepts that are either too abstract to understand or too detailed for early phase design exploration. This paper explores the uses of generative pre-trained transformers (GPT) for natural language design concept generation. Our experiments involve the use of GPT-2 and GPT-3 for different creative reasonings in design tasks. Both show reasonably good performance for verbal design concept generation.