# BikeBench: A Bicycle Design Benchmark for Generative Models with Objectives and Constraints

arXiv: https://arxiv.org/abs/2508.00830 · HTML: https://mlsanity.com/paper/2508.00830/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 98 references that are in the map.
- First posted on arXiv 25 May 2025.

## Builds on (most cited first, 30 of 40)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [ABC: A Big CAD Model Dataset For Geometric Deep Learning](https://mlsanity.com/paper/1812.06216/) (2018)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [DeepCAD: A Deep Generative Network for Computer-Aided Design Models](https://mlsanity.com/paper/2105.09492/) (2021)
- [Recent Trends in Deep Learning Based Natural Language Processing](https://mlsanity.com/paper/1708.02709/) (2017)
- [pymoo: Multi-objective Optimization in Python](https://mlsanity.com/paper/2002.04504/) (2020)
- [SciCode: A Research Coding Benchmark Curated by Scientists](https://mlsanity.com/paper/2407.13168/) (2024)
- [Deep Generative Models in Engineering Design: A Review](https://mlsanity.com/paper/2110.10863/) (2021)
- [Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization](https://mlsanity.com/paper/2202.08450/) (2022)
- [Deep Learning vs. Traditional Computer Vision](https://mlsanity.com/paper/1910.13796/) (2019)
- [Deep Generative Modeling for Mechanistic-based Learning and Design of Metamaterial Systems](https://mlsanity.com/paper/2006.15274/) (2020)
- [On the use of Artificial Neural Networks in Topology Optimisation](https://mlsanity.com/paper/2208.02563/) (2022)
- [DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentation](https://mlsanity.com/paper/2404.07917/) (2024)
- [A Comprehensive Survey on Word Representation Models: From Classical to State-Of-The-Art Word Representation Language Models](https://mlsanity.com/paper/2010.15036/) (2020)
- [BIKED: A Dataset for Computational Bicycle Design with Machine Learning Benchmarks](https://mlsanity.com/paper/2103.05844/) (2021)
- [Topology Optimization via Machine Learning and Deep Learning: A Review](https://mlsanity.com/paper/2210.10782/) (2022)
- [Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design](https://mlsanity.com/paper/2302.02913/) (2023)
- [DeepJEB: 3D Deep Learning-based Synthetic Jet Engine Bracket Dataset](https://mlsanity.com/paper/2406.09047/) (2024)
- [SimJEB: Simulated Jet Engine Bracket Dataset](https://mlsanity.com/paper/2105.03534/) (2021)
- [Ship-D: Ship Hull Dataset for Design Optimization using Machine Learning](https://mlsanity.com/paper/2305.08279/) (2023)
- [AircraftVerse: A Large-Scale Multimodal Dataset of Aerial Vehicle Designs](https://mlsanity.com/paper/2306.05562/) (2023)

## Built on by (most cited first, 5 of 5)

- [Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization](https://mlsanity.com/paper/2604.12290/) (2026)
- [Engineering Regression Without Real-Data Training: Domain Adaptation for Tabular Foundation Models Using Multi-Dataset Embeddings](https://mlsanity.com/paper/2603.04692/) (2026)
- [Do Diffusion Models Dream of Electric Planes? Discrete and Continuous Simulation-Based Inference for Aircraft Design](https://mlsanity.com/paper/2603.13284/) (2026)
- [GLUE: Coordinating Pre-Trained Generative Models for System-Level Design](https://mlsanity.com/paper/2512.19469/) (2025)
- [EngIntervene: Benchmarking Multimodal Engineering State Understanding and Design Intervention Reasoning](https://mlsanity.com/paper/2609.33261/) (2026)

## Abstract (arXiv)

We introduce BikeBench, an engineering design benchmark for evaluating generative models on problems with multiple real-world objectives and constraints. As generative AI's reach continues to grow, evaluating its capability to understand physical laws, human guidelines, and hard constraints grows increasingly important. Engineering product design lies at the intersection of these difficult tasks, providing new challenges for AI capabilities. BikeBench evaluates AI models' capabilities to generate bicycle designs that not only resemble the dataset, but meet specific performance objectives and constraints. To do so, BikeBench quantifies a variety of human-centered and multiphysics performance characteristics, such as aerodynamics, ergonomics, structural mechanics, human-rated usability, and similarity to subjective text or image prompts. Supporting the benchmark are several datasets of simulation results, a dataset of 10,000 human-rated bicycle assessments, and a synthetically generated dataset of 1.6M designs, each with a parametric, CAD/XML, SVG, and PNG representation. BikeBench is uniquely configured to evaluate tabular generative models, large language models (LLMs), design optimization, and hybrid algorithms side-by-side. Our experiments indicate that LLMs and tabular generative models fall short of hybrid GenAI+optimization algorithms in design quality, constraint satisfaction, and similarity scores, suggesting significant room for improvement. We hope that BikeBench, a first-of-its-kind benchmark, will help catalyze progress in generative AI for constrained multi-objective engineering design problems. We provide code, data, an interactive leaderboard, and other resources at https://github.com/Lyleregenwetter/BikeBench.