mlsanityData to 7 Oct 2026

OccamNets: Mitigating Dataset Bias by Favoring Simpler Hypotheses

arXiv 2204.02426 ↗First posted 5 Apr 2022cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Dataset bias and spurious correlations can significantly impair generalization in deep neural networks. Many prior efforts have addressed this problem using either alternative loss functions or sampling strategies that focus on rare patterns. We propose a new direction: modifying the network architecture to impose inductive biases that make the network robust to dataset bias. Specifically, we propose OccamNets, which are biased to favor simpler solutions by design. OccamNets have two inductive biases. First, they are biased to use as little network depth as needed for an individual example. Second, they are biased toward using fewer image locations for prediction. While OccamNets are biased toward simpler hypotheses, they can learn more complex hypotheses if necessary. In experiments, OccamNets outperform or rival state-of-the-art methods run on architectures that do not incorporate these inductive biases. Furthermore, we demonstrate that when the state-of-the-art debiasing methods are combined with OccamNets results further improve.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 67 of its 82 references that are in the map. First posted on arXiv 5 Apr 2022.

Builds on

67 of 82 references.

References · 67

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2017Mask R-CNNcited by 7,259
  8. 2016Densely Connected Convolutional Networkscited by 6,691
  9. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  12. 2014Going Deeper with Convolutionscited by 4,231
38 more
  1. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  2. 2019Searching for MobileNetV3cited by 1,984
  3. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  4. 2019Invariant Risk Minimizationcited by 1,832
  5. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  6. 2015Spatial Transformer Networkscited by 1,423
  7. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  8. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  9. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  10. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  11. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  12. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  13. 2018Mitigating Unwanted Biases with Adversarial Learningcited by 663
  14. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  15. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  16. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  17. 2014Deeply-Supervised Netscited by 364
  18. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  19. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  20. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  21. 2020Environment Inference for Invariant Learningcited by 306
  22. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  23. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  24. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 262
  25. 2018Learning Not to Learn: Training Deep Neural Networks with Biased Datacited by 255
  26. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  27. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  28. 2019RUBi: Reducing Unimodal Biases in Visual Question Answeringcited by 183
  29. 2020Characterising Bias in Compressed Modelscited by 162
  30. 2019REPAIR: Removing Representation Bias by Dataset Resamplingcited by 157
  31. 2020Counterfactual Samples Synthesizing for Robust Visual Question Answeringcited by 153
  32. 2018Gather-Excite: Exploiting Feature Context in Convolutional Neural Networkscited by 138
  33. 2019Training Neural Networks with Local Error Signalscited by 135
  34. 2020Learning from Failure: Training Debiased Classifier from Biased Classifiercited by 133
  35. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  36. 2018Overcoming Language Priors in Visual Question Answering with Adversarial Regularizationcited by 104
  37. 2020Don't Judge an Object by Its Context: Learning to Overcome Contextual Biascited by 91
  38. 2020Learning from others' mistakes: Avoiding dataset biases without modeling themcited by 83

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 1; 2026: 1.

11 papers.

  1. 2023Overcoming Simplicity Bias in Deep Networks using a Feature Sievecited by 29
  2. 2024What Variables Affect Out-of-Distribution Generalization in Pretrained Models?cited by 13
  3. 2025Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learningcited by 11
  4. 2024Efficient Bias Mitigation Without Privileged Informationcited by 8
  5. 2024DeNetDM: Debiasing by Network Depth Modulationcited by 6
  6. 2024BAdd: Bias Mitigation through Bias Additioncited by 5
  7. 2023Partition-and-Debias: Agnostic Biases Mitigation via A Mixture of Biases-Specific Expertscited by 3
  8. 2024Model Debiasing by Learnable Data Augmentationcited by 1
  9. 2024INSIGHT: Explainable Weakly-Supervised Medical Image Analysiscited by 1
  10. 2023Implicit Visual Bias Mitigation by Posterior Estimate Sharpening of a Bayesian Neural Networkcited by 0
  11. 2026Bias In, Bias Out? Finding Unbiased Subnetworks in Vanilla Modelscited by 0