# Generative Transformers for Design Concept Generation

arXiv: https://arxiv.org/abs/2211.03468 · HTML: https://mlsanity.com/paper/2211.03468/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 122 references that are in the map.
- First posted on arXiv 7 Nov 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [Deep Generative Models in Engineering Design: A Review](https://mlsanity.com/paper/2110.10863/) (2021)
- [Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses](https://mlsanity.com/paper/1708.07149/) (2017)
- [Few-shot Natural Language Generation for Task-Oriented Dialog](https://mlsanity.com/paper/2002.12328/) (2020)
- [TopologyGAN: Topology Optimization Using Generative Adversarial Networks Based on Physical Fields Over the Initial Domain](https://mlsanity.com/paper/2003.04685/) (2020)
- [Putting GPT-3's Creativity to the (Alternative Uses) Test](https://mlsanity.com/paper/2206.08932/) (2022)
- [Paraphrase Generation with Deep Reinforcement Learning](https://mlsanity.com/paper/1711.00279/) (2017)
- [Patent Claim Generation by Fine-Tuning OpenAI GPT-2](https://mlsanity.com/paper/1907.02052/) (2019)
- [Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models](https://mlsanity.com/paper/2002.05058/) (2020)
- [Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet](https://mlsanity.com/paper/2102.08036/) (2021)
- [Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation](https://mlsanity.com/paper/1805.08191/) (2018)
- [Data-Driven Design-by-Analogy: State of the Art and Future Directions](https://mlsanity.com/paper/2106.01592/) (2021)
- [Engineering Knowledge Graph from Patent Database](https://mlsanity.com/paper/2106.06739/) (2021)
- [Natural Language Processing in-and-for Design Research](https://mlsanity.com/paper/2111.13827/) (2021)
- [Data-Driven Innovation: What Is It](https://mlsanity.com/paper/2201.08184/) (2022)
- [Guiding Data-Driven Design Ideation by Knowledge Distance](https://mlsanity.com/paper/2210.10104/) (2022)
- [Design Representation as Semantic Networks](https://mlsanity.com/paper/2210.11533/) (2022)

## Built on by (most cited first, 21 of 21)

- [DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentation](https://mlsanity.com/paper/2404.07917/) (2024)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design](https://mlsanity.com/paper/2302.02913/) (2023)
- [Large Language Model Agent as a Mechanical Designer](https://mlsanity.com/paper/2404.17525/) (2024)
- [Conceptual Design Generation Using Large Language Models](https://mlsanity.com/paper/2306.01779/) (2023)
- [AutoTRIZ: Automating Engineering Innovation with TRIZ and Large Language Models](https://mlsanity.com/paper/2403.13002/) (2024)
- [Large Language Models for Combinatorial Optimization of Design Structure Matrix](https://mlsanity.com/paper/2411.12571/) (2024)
- [Toward Artificial Empathy for Human-Centered Design: A Framework](https://mlsanity.com/paper/2303.10583/) (2023)
- [Opportunities for Large Language Models and Discourse in Engineering Design](https://mlsanity.com/paper/2306.09169/) (2023)
- [Large Language Model for Patent Concept Generation](https://mlsanity.com/paper/2409.00092/) (2024)
- [A Novel Idea Generation Tool using a Structured Conversational AI (CAI) System](https://mlsanity.com/paper/2409.05747/) (2024)
- [Inspired by AI? A Novel Generative AI System To Assist Conceptual Automotive Design](https://mlsanity.com/paper/2407.11991/) (2024)
- [The Innovation Paradox: Concept Space Expansion with Diminishing Originality and the Promise of Creative AI](https://mlsanity.com/paper/2303.13300/) (2023)
- [Supermind Ideator: Exploring generative AI to support creative problem-solving](https://mlsanity.com/paper/2311.01937/) (2023)
- [Exploring the Capabilities of Large Language Models for Generating Diverse Design Solutions](https://mlsanity.com/paper/2405.02345/) (2024)
- [Evaluating Large Language Models for Material Selection](https://mlsanity.com/paper/2405.03695/) (2024)
- [Human-in-the-Loop: Quantitative Evaluation of 3D Models Generation by Large Language Models](https://mlsanity.com/paper/2509.07010/) (2025)
- [MSEval: A Dataset for Material Selection in Conceptual Design to Evaluate Algorithmic Models](https://mlsanity.com/paper/2407.09719/) (2024)
- [Large Language Models for Combinatorial Optimization of Design Structure Matrix](https://mlsanity.com/paper/2506.09749/) (2025)
- [Intelligent Design 4.0: Paradigm Evolution Toward the Agentic AI Era](https://mlsanity.com/paper/2506.09755/) (2025)
- [RePaint-Enhanced Conditional Diffusion Model for Parametric Engineering Designs under Performance and Parameter Constraints](https://mlsanity.com/paper/2602.00384/) (2026)

## Abstract (arXiv)

Generating novel and useful concepts is essential during the early design stage to explore a large variety of design opportunities, which usually requires advanced design thinking ability and a wide range of knowledge from designers. Growing works on computer-aided tools have explored the retrieval of knowledge and heuristics from design data. However, they only provide stimuli to inspire designers from limited aspects. This study explores the recent advance of the natural language generation (NLG) technique in the artificial intelligence (AI) field to automate the early-stage design concept generation. Specifically, a novel approach utilizing the generative pre-trained transformer (GPT) is proposed to leverage the knowledge and reasoning from textual data and transform them into new concepts in understandable language. Three concept generation tasks are defined to leverage different knowledge and reasoning: domain knowledge synthesis, problem-driven synthesis, and analogy-driven synthesis. The experiments with both human and data-driven evaluation show good performance in generating novel and useful concepts.