# Automated Synthetic-to-Real Generalization

arXiv: https://arxiv.org/abs/2007.06965 · HTML: https://mlsanity.com/paper/2007.06965/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2020.

## Built on by (most cited first, 30 of 35)

- [Domain Generalization: A Survey](https://mlsanity.com/paper/2103.02503/) (2021)
- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Generalize then Adapt: Source-Free Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2108.11249/) (2021)
- [WildNet: Learning Domain Generalized Semantic Segmentation from the Wild](https://mlsanity.com/paper/2204.01446/) (2022)
- [Label-Free Liver Tumor Segmentation](https://mlsanity.com/paper/2303.14869/) (2023)
- [Pin the Memory: Learning to Generalize Semantic Segmentation](https://mlsanity.com/paper/2204.03609/) (2022)
- [Source-Free Open Compound Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/2106.03422/) (2021)
- [PASTA: Proportional Amplitude Spectrum Training Augmentation for Syn-to-Real Domain Generalization](https://mlsanity.com/paper/2212.00979/) (2022)
- [Contrastive Syn-to-Real Generalization](https://mlsanity.com/paper/2104.02290/) (2021)
- [Informing Geometric Deep Learning with Electronic Interactions to Accelerate Quantum Chemistry](https://mlsanity.com/paper/2105.14655/) (2021)
- [Behavioral Intention Prediction in Driving Scenes: A Survey](https://mlsanity.com/paper/2211.00385/) (2022)
- [WEDGE: Web-Image Assisted Domain Generalization for Semantic Segmentation](https://mlsanity.com/paper/2109.14196/) (2021)
- [Texture Learning Domain Randomization for Domain Generalized Segmentation](https://mlsanity.com/paper/2303.11546/) (2023)
- [Domain Generalization via Balancing Training Difficulty and Model Capability](https://mlsanity.com/paper/2309.00844/) (2023)
- [Semantic Self-adaptation: Enhancing Generalization with a Single Sample](https://mlsanity.com/paper/2208.05788/) (2022)
- [Can We Solve 3D Vision Tasks Starting from A 2D Vision Transformer?](https://mlsanity.com/paper/2209.07026/) (2022)
- [Learning to Generalize Provably in Learning to Optimize](https://mlsanity.com/paper/2302.11085/) (2023)
- [Exposing Semantic Segmentation Failures via Maximum Discrepancy Competition](https://mlsanity.com/paper/2103.00259/) (2021)
- [Reducing the Amount of Real World Data for Object Detector Training with Synthetic Data](https://mlsanity.com/paper/2202.00632/) (2022)
- [M-L2O: Towards Generalizable Learning-to-Optimize by Test-Time Fast Self-Adaptation](https://mlsanity.com/paper/2303.00039/) (2023)
- [Improving the Effectiveness of Deep Generative Data](https://mlsanity.com/paper/2311.03959/) (2023)
- [False Negative Reduction in Semantic Segmentation under Domain Shift using Depth Estimation](https://mlsanity.com/paper/2207.03513/) (2022)
- [Weighted Risk Invariance: Domain Generalization under Invariant Feature Shift](https://mlsanity.com/paper/2407.18428/) (2024)
- [Learning Data-Efficient and Generalizable Neural Operators via Fundamental Physics Knowledge](https://mlsanity.com/paper/2602.15184/) (2026)
- [Exploration of Dark Chemical Genomics Space via Portal Learning: Applied to Targeting the Undruggable Genome and COVID-19 Anti-Infective Polypharmacology](https://mlsanity.com/paper/2111.14283/) (2021)
- [Optimizer Amalgamation](https://mlsanity.com/paper/2203.06474/) (2022)
- [GCISG: Guided Causal Invariant Learning for Improved Syn-to-real Generalization](https://mlsanity.com/paper/2208.10024/) (2022)
- [Adversarial training of Keyword Spotting to Minimize TTS Data Overfitting](https://mlsanity.com/paper/2408.10463/) (2024)
- [Make Optimization Once and for All with Fine-grained Guidance](https://mlsanity.com/paper/2503.11462/) (2025)
- [Geometric Synthesis: A Free lunch for Large-scale Palmprint Recognition Model Pretraining](https://mlsanity.com/paper/2203.05703/) (2022)

## Abstract (arXiv)

Models trained on synthetic images often face degraded generalization to real data. As a convention, these models are often initialized with ImageNet pre-trained representation. Yet the role of ImageNet knowledge is seldom discussed despite common practices that leverage this knowledge to maintain the generalization ability. An example is the careful hand-tuning of early stopping and layer-wise learning rates, which is shown to improve synthetic-to-real generalization but is also laborious and heuristic. In this work, we explicitly encourage the synthetically trained model to maintain similar representations with the ImageNet pre-trained model, and propose a textitlearning-to-optimize (L2O) strategy to automate the selection of layer-wise learning rates. We demonstrate that the proposed framework can significantly improve the synthetic-to-real generalization performance without seeing and training on real data, while also benefiting downstream tasks such as domain adaptation. Code is available at: https://github.com/NVlabs/ASG.