# Synth4Seg -- Learning Defect Data Synthesis for Defect Segmentation using Bi-level Optimization

arXiv: https://arxiv.org/abs/2410.18490 · HTML: https://mlsanity.com/paper/2410.18490/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 62 references that are in the map.
- First posted on arXiv 24 Oct 2024.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Towards Total Recall in Industrial Anomaly Detection](https://mlsanity.com/paper/2106.08265/) (2021)
- [Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations](https://mlsanity.com/paper/1707.03237/) (2017)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Bilevel Programming for Hyperparameter Optimization and Meta-Learning](https://mlsanity.com/paper/1806.04910/) (2018)
- [Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddings](https://mlsanity.com/paper/1911.02357/) (2019)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Fast AutoAugment](https://mlsanity.com/paper/1905.00397/) (2019)
- [Forward and Reverse Gradient-Based Hyperparameter Optimization](https://mlsanity.com/paper/1703.01785/) (2017)
- [TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation](https://mlsanity.com/paper/2103.10158/) (2021)
- [Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection](https://mlsanity.com/paper/1708.01642/) (2017)
- [Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules](https://mlsanity.com/paper/1905.05393/) (2019)
- [A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applications](https://mlsanity.com/paper/1705.06270/) (2017)
- [Adversarial AutoAugment](https://mlsanity.com/paper/1912.11188/) (2019)
- [Modeling Visual Context is Key to Augmenting Object Detection Datasets](https://mlsanity.com/paper/1807.07428/) (2018)
- [Defect-GAN: High-Fidelity Defect Synthesis for Automated Defect Inspection](https://mlsanity.com/paper/2103.15158/) (2021)
- [Deep Learning for Unsupervised Anomaly Localization in Industrial Images: A Survey](https://mlsanity.com/paper/2207.10298/) (2022)
- [Learning to Generate Synthetic Data via Compositing](https://mlsanity.com/paper/1904.05475/) (2019)
- [VISION Datasets: A Benchmark for Vision-based InduStrial InspectiON](https://mlsanity.com/paper/2306.07890/) (2023)
- [MetaAugment: Sample-Aware Data Augmentation Policy Learning](https://mlsanity.com/paper/2012.12076/) (2020)

## Built on by (most cited first, 3 of 3)

- [STAGE: Segmentation-oriented Industrial Anomaly Synthesis via Graded Diffusion with Explicit Mask Alignment](https://mlsanity.com/paper/2509.06693/) (2025)
- [Natural Hypergradient Descent: Algorithm Design, Convergence Analysis, and Parallel Implementation](https://mlsanity.com/paper/2602.10905/) (2026)
- [SARD: Segmentation-Aware Anomaly Synthesis via Region-Constrained Diffusion with Discriminative Mask Guidance](https://mlsanity.com/paper/2508.03143/) (2025)

## Abstract (arXiv)

Defect segmentation is crucial for quality control in advanced manufacturing, yet data scarcity poses challenges for state-of-the-art supervised deep learning. Synthetic defect data generation is a popular approach for mitigating data challenges. However, many current methods simply generate defects following a fixed set of rules, which may not directly relate to downstream task performance. This can lead to suboptimal performance and may even hinder the downstream task. To solve this problem, we leverage a novel bi-level optimization-based synthetic defect data generation framework. We use an online synthetic defect generation module grounded in the commonly-used Cut&Paste framework, and adopt an efficient gradient-based optimization algorithm to solve the bi-level optimization problem. We achieve simultaneous training of the defect segmentation network, and learn various parameters of the data synthesis module by maximizing the validation performance of the trained defect segmentation network. Our experimental results on benchmark datasets under limited data settings show that the proposed bi-level optimization method can be used for learning the most effective locations for pasting synthetic defects thereby improving the segmentation performance by up to 18.3% when compared to pasting defects at random locations. We also demonstrate up to 2.6% performance gain by learning the importance weights for different augmentation-specific defect data sources when compared to giving equal importance to all the data sources.