mlsanityData to 7 Oct 2026

Why do classifier accuracies show linear trends under distribution shift?

arXiv 2012.15483 ↗First posted 31 Dec 2020cs.LG stat.MLcited by 21

Not on the map. Why?

Abstract (arXiv)

Recent studies of generalization in deep learning have observed a puzzling trend: accuracies of models on one data distribution are approximately linear functions of the accuracies on another distribution. We explain this trend under an intuitive assumption on model similarity, which was verified empirically in prior work. More precisely, we assume the probability that two models agree in their predictions is higher than what we can infer from their accuracy levels alone. Then, we show that a linear trend must occur when evaluating models on two distributions unless the size of the distribution shift is large. This work emphasizes the value of understanding model similarity, which can have an impact on the generalization and robustness of classification models.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 6; 2024: 3; 2025: 1; 2026: 1.

21 papers.

Sort:

By area:

  1. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  2. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  3. 2023A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinkingcited by 85
  4. 2021The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuningcited by 74
  5. 2022When does dough become a bagel? Analyzing the remaining mistakes on ImageNetcited by 60
  6. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 46
  7. 2022ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasetscited by 44
  8. 2022Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regressioncited by 22
  9. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 20
  10. 2023Pathologies of Predictive Diversity in Deep Ensemblescited by 16
  11. 2021Patterns, predictions, and actions: A story about machine learningcited by 14
  12. 2022VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectivescited by 12
9 more
  1. 2023Test-Time Adaptation Induces Stronger Accuracy and Agreement-on-the-Linecited by 11
  2. 2023A Holistic Assessment of the Reliability of Machine Learning Systemscited by 9
  3. 2023DC4L: Distribution Shift Recovery via Data-Driven Control for Deep Learning Modelscited by 7
  4. 2024Don't Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budgetcited by 7
  5. 2023Automated Classification of Model Errors on ImageNetcited by 6
  6. 2022Understanding out-of-distribution accuracies through quantifying difficulty of test samplescited by 3
  7. 2022Monotonic Risk Relationships under Distribution Shifts for Regularized Risk Minimizationcited by 2
  8. 2026Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similaritycited by 1
  9. 2025Entropy-regularized Gradient Estimators for Approximate Bayesian Inferencecited by 0