mlsanityData to 14 Oct 2026

A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?

arXiv 2108.11018 ↗First posted 25 Aug 2021cs.LG cs.CVcited by 15

Not on the map. Why?

Abstract (arXiv)

Synthetic-to-real transfer learning is a framework in which a synthetically generated dataset is used to pre-train a model to improve its performance on real vision tasks. The most significant advantage of using synthetic images is that the ground-truth labels are automatically available, enabling unlimited expansion of the data size without human cost. However, synthetic data may have a huge domain gap, in which case increasing the data size does not improve the performance. How can we know that? In this study, we derive a simple scaling law that predicts the performance from the amount of pre-training data. By estimating the parameters of the law, we can judge whether we should increase the data or change the setting of image synthesis. Further, we analyze the theory of transfer learning by considering learning dynamics and confirm that the derived generalization bound is consistent with our empirical findings. We empirically validated our scaling law on various experimental settings of benchmark tasks, model sizes, and complexities of synthetic images.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 57 of its 71 references that are in the map. First posted on arXiv 25 Aug 2021.

Builds on

57 of 71 references.

References · 57

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  8. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  10. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  11. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  12. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
38 more
  1. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  2. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  3. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  4. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  5. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  6. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 687
  7. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  8. 2017Exploring Generalization in Deep Learningcited by 600
  9. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  10. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  11. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  12. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  13. 2019YOLACT: Real-time Instance Segmentationcited by 443
  14. 2018Rethinking ImageNet Pre-trainingcited by 377
  15. 2021Explaining Neural Scaling Lawscited by 363
  16. 2020Rethinking Pre-training and Self-trainingcited by 352
  17. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  18. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  19. 2018Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomizationcited by 294
  20. 2021Scaling Laws for Transfercited by 272
  21. 2020Few-Shot Learning via Learning the Representation, Provablycited by 214
  22. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  23. 2015The Benefit of Multitask Representation Learningcited by 195
  24. 2020BOP Challenge 2020 on 6D Object Localizationcited by 193
  25. 2016What makes ImageNet good for transfer learning?cited by 192
  26. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  27. 2015Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Viewscited by 94
  28. 2021Learning Curve Theorycited by 85
  29. 2020How Useful is Self-Supervised Pretraining for Visual Tasks?cited by 80
  30. 2019BlenderProccited by 77
  31. 2019Photorealistic Image Synthesis for Object Instance Detectioncited by 66
  32. 2020Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generationcited by 58
  33. 2019Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margincited by 56
  34. 2020A Neural Scaling Law from the Dimension of the Data Manifoldcited by 52
  35. 2017Synthesizing Training Data for Object Detection in Indoor Scenescited by 48
  36. 2021f-Domain-Adversarial Learning: Theory and Algorithmscited by 47
  37. 2020Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regimecited by 35
  38. 2021Contrastive Syn-to-Real Generalizationcited by 32

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 3; 2024: 5; 2025: 2; 2026: 3.

15 papers.

Sort:

By area:

  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  2. 2023Emergent and Predictable Memorization in Large Language Modelscited by 148
  3. 2025Scaling Laws for Optimal Data Mixturescited by 52
  4. 2024Scaling Laws for Downstream Task Performance of Large Language Modelscited by 49
  5. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 27
  6. 2021Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Datacited by 26
  7. 2025(Mis)Fitting: A Survey of Scaling Lawscited by 26
  8. 2023An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibrationcited by 5
  9. 2024Advancing Extrapolative Predictions of Material Properties through Learning to Learncited by 2
  10. 2024Scaling Law of Sim2Real Transfer Learning in Expanding Computational Materials Databases for Real-World Predictionscited by 2
  11. 2022Scalable Modular Synthetic Data Generation for Advancing Aerial Autonomycited by 1
  12. 2024Analyzing and reducing the synthetic-to-real transfer gap in Music Information Retrieval: the task of automatic drum transcriptioncited by 1
3 more
  1. 2026Provable Target Sample Complexity Improvements as Pre-Trained Models Scalecited by 0
  2. 2026Economics of Human and AI Collaboration: When is Partial Automation More Attractive than Full Automation?cited by 0
  3. 2026Provable Data Scaling Law for Meta Learning via Complexity Minimizationcited by 0