mlsanityData to 7 Oct 2026

Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Data

arXiv 2112.00054 ↗First posted 30 Nov 2021cs.CV cs.LGcited by 26

On the map: Synthetic data and self-improvement for LLMs → Self-improving MLLMs and synthetic data

Abstract (arXiv)

Pre-training models on Imagenet or other massive datasets of real images has led to major advances in computer vision, albeit accompanied with shortcomings related to curation cost, privacy, usage rights, and ethical issues. In this paper, for the first time, we study the transferability of pre-trained models based on synthetic data generated by graphics simulators to downstream tasks from very different domains. In using such synthetic data for pre-training, we find that downstream performance on different tasks are favored by different configurations of simulation parameters (e.g. lighting, object pose, backgrounds, etc.), and that there is no one-size-fits-all solution. It is thus better to tailor synthetic pre-training data to a specific downstream task, for best performance. We introduce Task2Sim, a unified model mapping downstream task representations to optimal simulation parameters to generate synthetic pre-training data for them. Task2Sim learns this mapping by training to find the set of best parameters on a set of "seen" tasks. Once trained, it can then be used to predict best simulation parameters for novel "unseen" tasks in one shot, without requiring additional training. Given a budget in number of images per class, our extensive experiments with 20 diverse downstream tasks show Task2Sim's task-adaptive pre-training data results in significantly better downstream performance than non-adaptively choosing simulation parameters on both seen and unseen tasks. It is even competitive with pre-training on real images from Imagenet.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Self-improving MLLMs and synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 67 of its 85 references that are in the map. First posted on arXiv 30 Nov 2021.

Builds on

67 of 85 references.

References · 67

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  7. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  11. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  12. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
38 more
  1. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  2. 2017Adversarial Discriminative Domain Adaptationcited by 1,309
  3. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  4. 2017CyCADA: Cycle-Consistent Adversarial Domain Adaptationcited by 1,079
  5. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  6. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  7. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  8. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  9. 2016Playing for Data: Ground Truth from Computer Gamescited by 816
  10. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  11. 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 657
  12. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  13. 2016Self-critical Sequence Training for Image Captioningcited by 544
  14. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  15. 2021Dynamic Neural Networks: A Surveycited by 405
  16. 2016AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classificationcited by 343
  17. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  18. 2016Learning from Simulated and Unsupervised Images through Adversarial Trainingcited by 287
  19. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  20. 2019Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Datacited by 245
  21. 2016Using Deep Learning for Image-Based Plant Disease Detectioncited by 216
  22. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  23. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 185
  24. 2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 166
  25. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 159
  26. 2017Convolutional Networks with Adaptive Inference Graphscited by 147
  27. 2018AdaFrame: Adaptive Frame Selection for Fast Video Recognitioncited by 134
  28. 2019Meta-Sim: Learning to Generate Synthetic Datasetscited by 128
  29. 2017Domain Adaptation for Visual Applications: A Comprehensive Surveycited by 127
  30. 2020Differential Treatment for Stuff and Things: A Simple Unsupervised Domain Adaptation Method for Semantic Segmentationcited by 126
  31. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  32. 2015Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Viewscited by 94
  33. 2018Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Datacited by 94
  34. 2021Learning to See by Looking at Noisecited by 92
  35. 2020AR-Net: Adaptive Frame Resolution for Efficient Action Recognitioncited by 88
  36. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 88
  37. 2016Beyond Sharing Weights for Deep Domain Adaptationcited by 80
  38. 2021A Broad Study on the Transferability of Visual Representations with Contrastive Learningcited by 78

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 6; 2024: 9; 2025: 6; 2026: 2.

26 papers.

Sort:

By area:

  1. 2024Position: Graph Foundation Models are Already Herecited by 84
  2. 2022PASTA: Proportional Amplitude Spectrum Training Augmentation for Syn-to-Real Domain Generalizationcited by 38
  3. 2023Going Beyond Nouns With Vision & Language Models Using Synthetic Datacited by 33
  4. 2023PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learningcited by 31
  5. 2023ExPT: Synthetic Pretraining for Few-Shot Experimental Designcited by 22
  6. 2024Synth2: Boosting Visual-Language Models with Synthetic Captions and Image Embeddingscited by 18
  7. 2022PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Datacited by 14
  8. 2024Can LLMs Replace Economic Choice Prediction Labs? The Case of Language-based Persuasion Gamescited by 12
  9. 2022Progressive Transformation Learning for Leveraging Virtual Images in Trainingcited by 10
  10. 2025CAMeL: Cross-modality Adaptive Meta-Learning for Text-based Person Retrievalcited by 6
  11. 2023M3Act: Learning from Synthetic Human Group Activitiescited by 5
  12. 2023Model Synthesis for Zero-Shot Model Attributioncited by 5
14 more
  1. 2023Camouflaged Image Synthesis Is All You Need to Boost Camouflaged Detectioncited by 5
  2. 2024BehAVE: Behaviour Alignment of Video Game Encodingscited by 5
  3. 2024Diversifying Human Pose in Synthetic Data for Aerial-view Human Detectioncited by 5
  4. 2024Learning from Synthetic Data for Visual Groundingcited by 4
  5. 2024Weak-to-Strong Compositional Learning from Generative Models for Language-based Object Detectioncited by 3
  6. 2026Scale Dependent Data Duplicationcited by 3
  7. 2025SoccerSynth-Detection: A Synthetic Dataset for Soccer Player Detectioncited by 2
  8. 2025Beyond Real Data: Synthetic Data through the Lens of Regularizationcited by 2
  9. 2025Action-guided generation of 3D functionality segmentation datacited by 2
  10. 2024Learning Conditional Invariances through Non-Commutativitycited by 1
  11. 2024Data augmentation with automated machine learning: approaches and performance comparison with classical data augmentation methodscited by 1
  12. 2026GMAIL: Generative Modality Alignment for generated Image Learningcited by 1
  13. 2025Industrial Synthetic Segment Pre-trainingcited by 0
  14. 2025Symmetria: A Synthetic Dataset for Learning in Point Cloudscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better
  2. 2024Scaling Backwards: Minimal Synthetic Pre-training?
  3. 2023Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models
  4. 2024Personalized Representation from Personalized Generation
  5. 2025Understanding Trade offs When Conditioning Synthetic Data
  6. 2024Mind the Gap Between Synthetic and Real: Utilizing Transfer Learning to Probe the Boundaries of Stable Diffusion Generated Data
  7. 2026When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators
  8. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clones