mlsanityData to 7 Oct 2026

Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?

arXiv 2106.02890 ↗First posted 5 Jun 2021cs.LG stat.MLcited by 75

Not on the map. Why?

Abstract (arXiv)

Can models with particular structure avoid being biased towards spurious correlation in out-of-distribution (OOD) generalization? Peters et al. (2016) provides a positive answer for linear cases. In this paper, we use a functional modular probing method to analyze deep model structures under OOD setting. We demonstrate that even in biased models (which focus on spurious correlation) there still exist unbiased functional subnetworks. Furthermore, we articulate and demonstrate the functional lottery ticket hypothesis: full network contains a subnetwork that can achieve better OOD performance. We then propose Modular Risk Minimization to solve the subnetwork selection problem. Our algorithm learns the subnetwork structure from a given dataset, and can be combined with any other OOD regularization methods. Experiments on various OOD generalization tasks corroborate the effectiveness of our method.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 63 of its 91 references that are in the map. First posted on arXiv 5 Jun 2021.

Builds on

63 of 91 references.

References · 63

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2016Wide Residual Networkscited by 3,057
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  5. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  8. 2019Invariant Risk Minimizationcited by 1,832
  9. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  10. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  11. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  12. 2020In Search of Lost Domain Generalizationcited by 1,042
38 more
  1. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  2. 2018Robustness May Be at Odds with Accuracycited by 873
  3. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  4. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  5. 2018Recognition in Terra Incognitacited by 655
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  7. 2013Domain Generalization via Invariant Feature Representationcited by 622
  8. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  9. 2018Rethinking the Value of Network Pruningcited by 595
  10. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  11. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 533
  12. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  13. 2015Neural Module Networkscited by 375
  14. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  15. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  16. 2017Unified Deep Supervised Domain Adaptation and Generalizationcited by 322
  17. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  18. 2020The Risks of Invariant Risk Minimizationcited by 262
  19. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 262
  20. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  21. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
  22. 2019Recurrent Independent Mechanismscited by 219
  23. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 206
  24. 2019Learning De-biased Representations with Biased Representationscited by 199
  25. 2020Invariant Risk Minimization Gamescited by 188
  26. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  27. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  28. 2019What Do Compressed Deep Neural Networks Forget?cited by 164
  29. 2020Characterising Bias in Compressed Modelscited by 162
  30. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  31. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  32. 2020Learning from Failure: Training Debiased Classifier from Biased Classifiercited by 133
  33. 2020Learning explanations that are hard to varycited by 132
  34. 2020Invariant Rationalizationcited by 131
  35. 2019Learning Robust Representations by Projecting Superficial Statistics Outcited by 123
  36. 2021Does Invariant Risk Minimization Capture Invariance?cited by 116
  37. 2021Counterfactual Generative Networkscited by 111
  38. 2020Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Maskscited by 97

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 21; 2023: 24; 2024: 15; 2025: 3; 2026: 6.

75 papers.

Sort:

By area:

  1. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 467
  2. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  3. 2022Handling Distribution Shifts on Graphs: An Invariance Perspectivecited by 163
  4. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  5. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  6. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  7. 2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
  8. 2022Agree to Disagree: Diversity through Disagreement for Better Transferabilitycited by 69
  9. 2021A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustnesscited by 62
  10. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  11. 2023Model Agnostic Sample Reweighting for Out-of-Distribution Learningcited by 43
  12. 2021Out-of-distribution Generalization in the Presence of Nuisance-Induced Spurious Correlationscited by 41
38 more
  1. 2022Fortuitous Forgetting in Connectionist Networkscited by 39
  2. 2021NAS-OoD: Neural Architecture Search for Out-of-Distribution Generalizationcited by 28
  3. 2023Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!cited by 27
  4. 2022Sparse Double Descent: Where Network Pruning Aggravates Overfittingcited by 26
  5. 2023Spurious Feature Diversification Improves Out-of-distribution Generalizationcited by 26
  6. 2023Distribution Shift Inversion for Out-of-Distribution Predictioncited by 23
  7. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  8. 2023DomainDrop: Suppressing Domain-Sensitive Channels for Domain Generalizationcited by 21
  9. 2024Uncertainty-Calibrated Test-Time Model Adaptation without Forgettingcited by 20
  10. 2021Learning Transferable Parameters for Unsupervised Domain Adaptationcited by 19
  11. 2022Exploring Optimal Substructure for Out-of-distribution Generalization via Feature-targeted Model Pruningcited by 17
  12. 2022Training Debiased Subnetworks with Contrastive Weight Pruningcited by 16
  13. 2022Robust Lottery Tickets for Pre-trained Language Modelscited by 16
  14. 2023Towards Artificial General Intelligence (AGI) in the Internet of Things (IoT): Opportunities and Challengescited by 14
  15. 2025Stochastic Parameter Decompositioncited by 14
  16. 2023Winning Prize Comes from Losing Tickets: Improve Invariant Learning by Exploring Variant Parameters for Out-of-Distribution Generalizationcited by 12
  17. 2023Uncovering Intermediate Variables in Transformers using Circuit Probingcited by 12
  18. 2024SAFT: Towards Out-of-Distribution Generalization in Fine-Tuningcited by 12
  19. 2024Continual Learning on Graphs: A Surveycited by 11
  20. 2024Sheaf Discovery with Joint Computation Graph Pruning and Flexible Granularitycited by 11
  21. 2024FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusioncited by 11
  22. 2023What Is Missing in IRM Training and Evaluation? Challenges and Solutionscited by 10
  23. 2023Diverse Target and Contribution Scheduling for Domain Generalizationcited by 10
  24. 2023SFP: Spurious Feature-targeted Pruning for Out-of-Distribution Generalizationcited by 9
  25. 2023NeuroSurgeon: A Toolkit for Subnetwork Analysiscited by 9
  26. 2024Out of Spuriousity: Improving Robustness to Spurious Correlations without Group Annotationscited by 9
  27. 2022Self-supervised debiasing using low rank regularizationcited by 8
  28. 2023Moderately Distributional Exploration for Domain Generalizationcited by 8
  29. 2023Even Small Correlation and Diversity Shifts Pose Dataset-Bias Issuescited by 8
  30. 2022A Win-win Deal: Towards Sparse and Robust Pre-trained Language Modelscited by 7
  31. 2023Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Modelscited by 7
  32. 2024DeNetDM: Debiasing by Network Depth Modulationcited by 6
  33. 2024What Does Softmax Probability Tell Us about Classifiers Ranking Across Diverse Test Conditions?cited by 6
  34. 2022Data-Efficient Double-Win Lottery Tickets from Robust Pre-trainingcited by 5
  35. 2022On the Connection between Invariant Learning and Adversarial Training for Out-of-Distribution Generalizationcited by 5
  36. 2023Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Modelscited by 5
  37. 2025Shifting the Paradigm: A Diffeomorphism Between Time Series Data Manifolds for Achieving Shift-Invariancy in Deep Learningcited by 5
  38. 2026Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modelingcited by 5

50 most cited of 75 All 75 (JSON)