mlsanityData to 7 Oct 2026

Learning explanations that are hard to vary

arXiv 2009.00329 ↗First posted 1 Sep 2020cs.LG stat.MLcited by 132

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate the principle that `good explanations are hard to vary' in the context of deep learning. We show that averaging gradients across examples -- akin to a logical OR of patterns -- can favor memorization and `patchwork' solutions that sew together different strategies, instead of identifying invariances. To inspect this, we first formalize a notion of consistency for minima of the loss surface, which measures to what extent a minimum appears only when examples are pooled. We then propose and experimentally validate a simple alternative algorithm based on a logical AND, that focuses on invariances and prevents memorization in a set of real-world tasks. Finally, using a synthetic dataset with a clear distinction between invariant and spurious mechanisms, we dissect learning signals and compare this approach to well-established regularizers.

Key facts. Cited by 132 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 26; 2022: 32; 2023: 30; 2024: 21; 2025: 17; 2026: 6.

132 papers.

Sort:

By area:

  1. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  2. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 258
  3. 2021Gradient Matching for Domain Generalizationcited by 258
  4. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  5. 2021Towards Causal Representation Learningcited by 207
  6. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  7. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
  8. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  9. 2021Towards a Theoretical Framework of Out-of-Distribution Generalizationcited by 94
  10. 2021OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalizationcited by 86
  11. 2021Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalizationcited by 83
  12. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
38 more
  1. 2023Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcutscited by 63
  2. 2021Domain Generalization via Gradient Surgerycited by 60
  3. 2022Assaying Out-Of-Distribution Generalization in Transfer Learningcited by 53
  4. 2021SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalizationcited by 50
  5. 2021A Theory of Label Propagation for Subpopulation Shiftcited by 48
  6. 2021Active label cleaning for improved dataset quality under resource constraintscited by 44
  7. 2022Fortuitous Forgetting in Connectionist Networkscited by 39
  8. 2022Pareto Invariant Risk Minimization: Towards Mitigating the Optimization Dilemma in Out-of-Distribution Generalizationcited by 38
  9. 2023Understanding and Improving Feature Learning for Out-of-Distribution Generalizationcited by 35
  10. 2022Domain-invariant Feature Exploration for Domain Generalizationcited by 34
  11. 2023Neuro-Symbolic Continual Learning: Knowledge, Reasoning Shortcuts and Concept Rehearsalcited by 33
  12. 2022Provable Domain Generalization via Invariant-Feature Subspace Recoverycited by 32
  13. 2022Out-of-Distribution Representation Learning for Time Series Classificationcited by 30
  14. 2021Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalizationcited by 29
  15. 2022WOODS: Benchmarks for Out-of-Distribution Generalization in Time Seriescited by 29
  16. 2023Neuron Structure Modeling for Generalizable Remote Physiological Measurementcited by 29
  17. 2023Domain Generalization in Computational Pathology: Survey and Guidelinescited by 29
  18. 2022Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognitioncited by 28
  19. 2023Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalizationcited by 28
  20. 2021Linear unit-tests for invariance discoverycited by 27
  21. 2021An Information-theoretic Approach to Distribution Shiftscited by 25
  22. 2022Learning Invariant Visual Representations for Compositional Zero-Shot Learningcited by 24
  23. 2023DIVERSIFY: A General Framework for Time Series Out-of-distribution Detection and Generalizationcited by 24
  24. 2023Distribution Shift Inversion for Out-of-Distribution Predictioncited by 23
  25. 2022Gradient Masked Averaging for Federated Learningcited by 22
  26. 2023MADG: Margin-based Adversarial Learning for Domain Generalizationcited by 22
  27. 2022The Two Dimensions of Worst-case Training and the Integrated Effect for Out-of-domain Generalizationcited by 21
  28. 2022Causal Balancing for Domain Generalizationcited by 20
  29. 2022Localized Adversarial Domain Generalizationcited by 19
  30. 2021Model-Invariant State Abstractions for Model-Based Reinforcement Learningcited by 18
  31. 2022Generalizing to Evolving Domains with Latent Structure-Aware Sequential Autoencodercited by 18
  32. 2021Learning Representations that Support Robust Transfer of Predictorscited by 17
  33. 2022Improving Out-of-Distribution Generalization by Adversarial Training with Structured Priorscited by 17
  34. 2022Domain Generalization for Activity Recognition via Adaptive Feature Fusioncited by 16
  35. 2023Understanding Hessian Alignment for Domain Generalizationcited by 15
  36. 2022FL Games: A Federated Learning Framework for Distribution Shiftscited by 14
  37. 2023Improving Generalization in Visual Reinforcement Learning via Conflict-aware Gradient Agreement Augmentationcited by 13
  38. 2024Advancing Generalizable Remote Physiological Measurement through the Integration of Explicit and Implicit Prior Knowledgecited by 13

50 most cited of 132 All 132 (JSON)