# DepthBenchCAD: When Does Deeper Auditing Yield More Reliable Conclusions?

arXiv: https://arxiv.org/abs/2609.15122 · HTML: https://mlsanity.com/paper/2609.15122/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 14 Sep 2026.

## Builds on (most cited first, 28 of 28)

- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [ABC: A Big CAD Model Dataset For Geometric Deep Learning](https://mlsanity.com/paper/1812.06216/) (2018)
- [DeepCAD: A Deep Generative Network for Computer-Aided Design Models](https://mlsanity.com/paper/2105.09492/) (2021)
- [BSP-Net: Generating Compact Meshes via Binary Space Partitioning](https://mlsanity.com/paper/1911.06971/) (2019)
- [Learning Shape Abstractions by Assembling Volumetric Primitives](https://mlsanity.com/paper/1612.00404/) (2016)
- [CvxNet: Learnable Convex Decomposition](https://mlsanity.com/paper/1909.05736/) (2019)
- [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018)
- [With Little Power Comes Great Responsibility](https://mlsanity.com/paper/2010.06595/) (2020)
- [Superquadrics Revisited: Learning 3D Shape Parsing beyond Cuboids](https://mlsanity.com/paper/1904.09970/) (2019)
- [CSGNet: Neural Shape Parser for Constructive Solid Geometry](https://mlsanity.com/paper/1712.08290/) (2017)
- [Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks](https://mlsanity.com/paper/2103.10429/) (2021)
- [BrepGen: A B-rep Generative Diffusion Model with Structured Latent Geometry](https://mlsanity.com/paper/2401.15563/) (2024)
- [Supervised Fitting of Geometric Primitives to 3D Point Clouds](https://mlsanity.com/paper/1811.08988/) (2018)
- [JoinABLe: Learning Bottom-up Assembly of Parametric CAD Joints](https://mlsanity.com/paper/2111.12772/) (2021)
- [Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging](https://mlsanity.com/paper/1707.09861/) (2017)
- [CAD-Recode: Reverse Engineering CAD Code from Point Clouds](https://mlsanity.com/paper/2412.14042/) (2024)
- [CAPRI-Net: Learning Compact CAD Shapes with Adaptive Primitive Assembly](https://mlsanity.com/paper/2104.05652/) (2021)
- [Point2Cyl: Reverse Engineering 3D Objects from Point Clouds to Extrusion Cylinders](https://mlsanity.com/paper/2112.09329/) (2021)
- [SECAD-Net: Self-Supervised CAD Reconstruction by Learning Sketch-Extrude Operations](https://mlsanity.com/paper/2303.10613/) (2023)
- [HPNet: Deep Primitive Segmentation Using Hybrid Representations](https://mlsanity.com/paper/2105.10620/) (2021)
- [Self-Supervised Representation Learning for CAD](https://mlsanity.com/paper/2210.10807/) (2022)
- [CAD-Deform: Deformable Fitting of CAD Models to 3D Scans](https://mlsanity.com/paper/2007.11965/) (2020)
- [BenchCAD: A Comprehensive, Industry-Standard Benchmark for Programmatic CAD](https://mlsanity.com/paper/2605.10865/) (2026)
- [Text2CAD-Bench: A Benchmark for LLM-based Text-to-Parametric CAD Generation](https://mlsanity.com/paper/2605.18430/) (2026)
- [Implicit Conversion of Manifold B-Rep Solids by Neural Halfspace Representation](https://mlsanity.com/paper/2209.10191/) (2022)

## Abstract (arXiv)

Generative CAD models are expected to remain behaviorally correct after parameter edits, so increasing the number of edit checks is often treated as a direct route to more reliable evaluation. Under a fixed budget, however, auditing each program more thoroughly reduces the number of tasks and independent generations that can be evaluated, which can ultimately make model-level estimates less accurate. We study this phenomenon and the conditions under which it arises. We decompose behavioral evaluation into three evidence levels: task templates, stochastic generations, and within-program edits. We define an average failure risk that is invariant to audit depth, and combine three-level variance with measured execution costs to analyze the tradeoff between deeper edit auditing and broader independent coverage. Experiments across two CAD environments and five generation systems show that the value of deeper auditing depends on where evaluation uncertainty originates. When template heterogeneity or generation stochasticity dominates, additional edit checks can increase total estimation error; when within-program state variation is large and generation is expensive, deeper auditing is more valuable. Variance and cost estimates from calibration predict the direction of this change and provide a diagnostic basis for allocating evidence on held-out tasks. These results show that the thoroughness of program inspection can diverge from the reliability of model evaluation, and they help determine whether the next unit of budget should be spent on a new task, a new generation, or additional edit checks.