# Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design

arXiv: https://arxiv.org/abs/2302.02913 · HTML: https://mlsanity.com/paper/2302.02913/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 52 of its 134 references that are in the map.
- First posted on arXiv 6 Feb 2023.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 52)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [Towards Principled Methods for Training Generative Adversarial Networks](https://mlsanity.com/paper/1701.04862/) (2017)
- [Assessing Generative Models via Precision and Recall](https://mlsanity.com/paper/1806.00035/) (2018)
- [Are GANs Created Equal? A Large-Scale Study](https://mlsanity.com/paper/1711.10337/) (2017)
- [A Comprehensive Survey of Graph Embedding: Problems, Techniques and Applications](https://mlsanity.com/paper/1709.07604/) (2017)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [A Note on the Inception Score](https://mlsanity.com/paper/1801.01973/) (2018)
- [Pros and Cons of GAN Evaluation Measures](https://mlsanity.com/paper/1802.03446/) (2018)
- [Hyperbolic Image Embeddings](https://mlsanity.com/paper/1904.02239/) (2019)
- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [Classification Accuracy Score for Conditional Generative Models](https://mlsanity.com/paper/1905.10887/) (2019)
- [Pros and Cons of GAN Evaluation Measures: New Developments](https://mlsanity.com/paper/2103.09396/) (2021)

## Built on by (most cited first, 23 of 23)

- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [Elicitron: An LLM Agent-Based Simulation Framework for Design Requirements Elicitation](https://mlsanity.com/paper/2404.16045/) (2024)
- [FigGen: Text to Scientific Figure Generation](https://mlsanity.com/paper/2306.00800/) (2023)
- [Conceptual Design Generation Using Large Language Models](https://mlsanity.com/paper/2306.01779/) (2023)
- [Constraining Generative Models for Engineering Design with Negative Data](https://mlsanity.com/paper/2306.15166/) (2023)
- [ShipGen: A Diffusion Model for Parametric Ship Hull Generation with Multiple Objectives and Constraints](https://mlsanity.com/paper/2311.06315/) (2023)
- [AutoTRIZ: Automating Engineering Innovation with TRIZ and Large Language Models](https://mlsanity.com/paper/2403.13002/) (2024)
- [DATED: Guidelines for Creating Synthetic Datasets for Engineering Design Applications](https://mlsanity.com/paper/2305.09018/) (2023)
- [Opportunities for Large Language Models and Discourse in Engineering Design](https://mlsanity.com/paper/2306.09169/) (2023)
- [Prompting for products: Investigating design space exploration strategies for text-to-image generative models](https://mlsanity.com/paper/2408.03946/) (2024)
- [EngiBench: A Framework for Data-Driven Engineering Design Research](https://mlsanity.com/paper/2508.00831/) (2025)
- [MCD: A Model-Agnostic Counterfactual Search Method For Multi-modal Design Modifications](https://mlsanity.com/paper/2305.11308/) (2023)
- [From Cloud to Edge: Rethinking Generative AI for Low-Resource Design Challenges](https://mlsanity.com/paper/2402.12702/) (2024)
- [BikeBench: A Bicycle Design Benchmark for Generative Models with Objectives and Constraints](https://mlsanity.com/paper/2508.00830/) (2025)
- [Scorecards for Synthetic Medical Data Evaluation and Reporting](https://mlsanity.com/paper/2406.11143/) (2024)
- [CAD-Prompted Generative Models: A Pathway to Feasible and Novel Engineering Designs](https://mlsanity.com/paper/2407.08675/) (2024)
- [Exploring the Capabilities of Large Language Models for Generating Diverse Design Solutions](https://mlsanity.com/paper/2405.02345/) (2024)
- [GeoBiked: A Dataset with Geometric Features and Automated Labeling Techniques to Enable Deep Generative Models in Engineering Design](https://mlsanity.com/paper/2409.17045/) (2024)
- [Bridging Design Gaps: A Parametric Data Completion Approach With Graph Guided Diffusion Models](https://mlsanity.com/paper/2406.11934/) (2024)
- [Image2Gcode: Image-to-G-code Generation for Additive Manufacturing Using Diffusion-Transformer Model](https://mlsanity.com/paper/2511.20636/) (2025)
- [A Meta-Generation framework for Industrial System Generation](https://mlsanity.com/paper/2306.05123/) (2023)
- [GreenPlanner: Practical Floorplan Layout Generation via an Energy-Aware and Function-Feasible Generative Framework](https://mlsanity.com/paper/2512.00406/) (2025)
- [MAGNET: Manifold-Aware Graph Diffusion Network for Connectome Generation](https://mlsanity.com/paper/2610.17014/) (2026)

## Abstract (arXiv)

Deep generative models such as Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), Diffusion Models, and Transformers, have shown great promise in a variety of applications, including image and speech synthesis, natural language processing, and drug discovery. However, when applied to engineering design problems, evaluating the performance of these models can be challenging, as traditional statistical metrics based on likelihood may not fully capture the requirements of engineering applications. This paper doubles as a review and practical guide to evaluation metrics for deep generative models (DGMs) in engineering design. We first summarize the well-accepted `classic' evaluation metrics for deep generative models grounded in machine learning theory. Using case studies, we then highlight why these metrics seldom translate well to design problems but see frequent use due to the lack of established alternatives. Next, we curate a set of design-specific metrics which have been proposed across different research communities and can be used for evaluating deep generative models. These metrics focus on unique requirements in design and engineering, such as constraint satisfaction, functional performance, novelty, and conditioning. Throughout our discussion, we apply the metrics to models trained on simple-to-visualize 2-dimensional example problems. Finally, we evaluate four deep generative models on a bicycle frame design problem and structural topology generation problem. In particular, we showcase the use of proposed metrics to quantify performance target achievement, design novelty, and geometric constraints. We publicly release the code for the datasets, models, and metrics used throughout the paper at https://decode.mit.edu/projects/metrics/.