mlsanityData to 7 Oct 2026

NICO++: Towards Better Benchmarking for Domain Generalization

arXiv 2204.08040 ↗First posted 17 Apr 2022cs.CV cs.LGcited by 86

Not on the map. Why?

Abstract (arXiv)

Despite the remarkable performance that modern deep neural networks have achieved on independent and identically distributed (I.I.D.) data, they can crash under distribution shifts. Most current evaluation methods for domain generalization (DG) adopt the leave-one-out strategy as a compromise on the limited number of domains. We propose a large-scale benchmark with extensive labeled domains named NICO++ along with more rational evaluation methods for comprehensively evaluating DG algorithms. To evaluate DG datasets, we propose two metrics to quantify covariate shift and concept shift, respectively. Two novel generalization bounds from the perspective of data construction are proposed to prove that limited concept shift and significant covariate shift favor the evaluation capability for generalization. Through extensive experiments, NICO++ shows its superior evaluation capability compared with current DG datasets and its contribution in alleviating unfairness caused by the leak of oracle knowledge in model selection.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Builds on 74 of its 113 references that are in the map. First posted on arXiv 17 Apr 2022.

Builds on

74 of 113 references.

References · 74

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  10. 2019Invariant Risk Minimizationcited by 1,832
  11. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  12. 2019Natural Adversarial Examplescited by 1,401
38 more
  1. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  2. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  3. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  4. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,194
  5. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  6. 2020In Search of Lost Domain Generalizationcited by 1,042
  7. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  8. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 762
  9. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  10. 2021Domain Generalization: A Surveycited by 760
  11. 2018Recognition in Terra Incognitacited by 655
  12. 2013Domain Generalization via Invariant Feature Representationcited by 622
  13. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  14. 2021Domain Generalization with MixStylecited by 563
  15. 2018Generalizing to Unseen Domains via Adversarial Data Augmentationcited by 451
  16. 2019Domain Generalization by Solving Jigsaw Puzzlescited by 449
  17. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  18. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  19. 2020Self-Challenging Improves Cross-Domain Generalizationcited by 426
  20. 2017Functional Map of the Worldcited by 338
  21. 2019Bridging Theory and Algorithm for Domain Adaptationcited by 332
  22. 2018Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomizationcited by 292
  23. 2021A Fourier-based Framework for Domain Generalizationcited by 287
  24. 2018Generalizing Across Domains via Cross-Gradient Trainingcited by 268
  25. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 262
  26. 2015Domain Generalization for Object Recognition with Multi-task Autoencoderscited by 253
  27. 2019Episodic Training for Domain Generalizationcited by 245
  28. 2019Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Datacited by 245
  29. 2020Deep Domain-Adversarial Image Generation for Domain Generalisationcited by 231
  30. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 218
  31. 2019Causality matters in medical imagingcited by 193
  32. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  33. 2020Improve Unsupervised Domain Adaptation with Mixup Trainingcited by 185
  34. 2017Domain Generalization by Marginal Transfer Learningcited by 180
  35. 2019Domain Generalization Using a Mixture of Multiple Latent Domainscited by 170
  36. 2021Deep Stable Learning for Out-Of-Distribution Generalizationcited by 153
  37. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  38. 2018Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objectscited by 143

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 20; 2024: 22; 2025: 17; 2026: 14.

86 papers.

Sort:

By area:

  1. 2022GOOD: A Graph Out-of-Distribution Benchmarkcited by 140
  2. 2023Change is Hard: A Closer Look at Subpopulation Shiftcited by 119
  3. 2024LLM as Dataset Analyst: Subpopulation Structure Discovery with Large Language Modelcited by 43
  4. 2023Exploring One-shot Semi-supervised Federated Learning with A Pre-trained Diffusion Modelcited by 33
  5. 2023Feedback-guided Data Synthesis for Imbalanced Classificationcited by 33
  6. 2023Joint Learning of Label and Environment Causal Independence for Graph Out-of-Distribution Generalizationcited by 29
  7. 2023Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalizationcited by 28
  8. 2022Context-Aware Robust Fine-Tuningcited by 27
  9. 2023Flatness-Aware Minimization for Domain Generalizationcited by 27
  10. 2023FACTS: First Amplify Correlations and Then Slice to Discover Biascited by 26
  11. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  12. 2023Industrial Anomaly Detection with Domain Shift: A Real-world Dataset and Masked Multi-scale Reconstructioncited by 22
38 more
  1. 2023Rethinking the Evaluation Protocol of Domain Generalizationcited by 20
  2. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 20
  3. 2023FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clientscited by 18
  4. 2024FedDEO: Description-Enhanced One-Shot Federated Learning with Diffusion Modelscited by 17
  5. 2022Bootstrap Generalization Ability from Loss Landscape Perspectivecited by 16
  6. 2025CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Surveycited by 15
  7. 2022Meta-Causal Feature Learning for Out-of-Distribution Generalizationcited by 14
  8. 2022Domain Generalization -- A Causal Perspectivecited by 14
  9. 2025Handling Out-of-Distribution Data: A Surveycited by 14
  10. 2022Stable Learning via Sparse Variable Independencecited by 13
  11. 2024Compositional Risk Minimizationcited by 13
  12. 2022Measures of Information Reflect Memorization Patternscited by 12
  13. 2024Fairness and Bias Mitigation in Computer Vision: A Surveycited by 12
  14. 2025Instance-Level Composed Image Retrievalcited by 12
  15. 2023Beyond Distribution Shift: Spurious Features Through the Lens of Training Dynamicscited by 11
  16. 2023What Is Missing in IRM Training and Evaluation? Challenges and Solutionscited by 10
  17. 2023Graph Structure and Feature Extrapolation for Out-of-Distribution Generalizationcited by 10
  18. 2025One-Shot Federated Learning with Classifier-Free Diffusion Modelscited by 7
  19. 2023MoP-CLIP: A Mixture of Prompt-Tuned CLIP Models for Domain Incremental Learningcited by 6
  20. 2024DPStyler: Dynamic PromptStyler for Source-Free Domain Generalizationcited by 6
  21. 2024Grounding Stylistic Domain Generalization with Quantitative Domain Shift Measures and Synthetic Scene Imagescited by 6
  22. 2025Mitigating Clever Hans Strategies in Image Classifiers through Generating Counterexamplescited by 6
  23. 2023One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Modelscited by 5
  24. 2024SLIM: Spuriousness Mitigation with Minimal Human Annotationscited by 5
  25. 2024Comprehensive Attribution: Inherently Explainable Vision Model with Feature Detectorcited by 5
  26. 2024What could go wrong? Discovering and describing failure modes in computer visioncited by 5
  27. 2025ConceptScope: Characterizing Dataset Bias via Disentangled Visual Conceptscited by 5
  28. 2023Challenges and Opportunities in Improving Worst-Group Generalization in Presence of Spurious Featurescited by 4
  29. 2023Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shiftcited by 4
  30. 2024Reassessing the Validity of Spurious Correlations Benchmarkscited by 4
  31. 2024Crafting Distribution Shifts for Validation and Training in Single Source Domain Generalizationcited by 4
  32. 2024Composed Image Retrieval for Training-Free Domain Conversioncited by 4
  33. 2025Diverse Prototypical Ensembles Improve Robustness to Subpopulation Shiftcited by 4
  34. 2025The Impact of Coreset Selection on Spurious Correlations and Group Robustnesscited by 4
  35. 2022Learning Less Generalizable Patterns with an Asymmetrically Trained Double Classifier for Better Test-Time Adaptationcited by 3
  36. 2024DomainVerse: A Benchmark Towards Real-World Distribution Shifts For Tuning-Free Adaptive Domain Generalizationcited by 3
  37. 2024Disentangling Masked Autoencoders for Unsupervised Domain Generalizationcited by 3
  38. 2022Domain Decorrelation with Potential Energy Rankingcited by 2

50 most cited of 86 All 86 (JSON)