mlsanityData to 7 Oct 2026

Demystifying Disagreement-on-the-Line in High Dimensions

arXiv 2301.13371 ↗First posted 31 Jan 2023stat.ML cs.CV cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Evaluating the performance of machine learning models under distribution shift is challenging, especially when we only have unlabeled data from the shifted (target) domain, along with labeled data from the original (source) domain. Recent work suggests that the notion of disagreement, the degree to which two models trained with different randomness differ on the same input, is a key to tackle this problem. Experimentally, disagreement and prediction error have been shown to be strongly connected, which has been used to estimate model performance. Experiments have led to the discovery of the disagreement-on-the-line phenomenon, whereby the classification error under the target domain is often a linear function of the classification error under the source domain; and whenever this property holds, disagreement under the source and target domain follow the same linear relation. In this work, we develop a theoretical foundation for analyzing disagreement in high-dimensional random features regression; and study under what conditions the disagreement-on-the-line phenomenon occurs in our setting. Experiments on CIFAR-10-C, Tiny ImageNet-C, and Camelyon17 are consistent with our theory and support the universality of the theoretical findings.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 41 of its 128 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

41 of 128 references.

References · 41

  1. 2013Intriguing properties of neural networkscited by 5,114
  2. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  3. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  5. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  6. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  7. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  8. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  9. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  10. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  11. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  12. 2019Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imagingcited by 215
29 more
  1. 2019Linearized two-layers neural networks in high dimensioncited by 155
  2. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  3. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  4. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 141
  5. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  6. 2021Assessing Generalization of SGD via Disagreementcited by 116
  7. 2021Predicting with Confidence on Unseen Distributionscited by 116
  8. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  9. 2020Are Labels Always Necessary for Classifier Accuracy Evaluation?cited by 104
  10. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  11. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  12. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  13. 2020Understanding Double Descent Requires a Fine-Grained Bias-Variance Decompositioncited by 75
  14. 2022Universality of empirical risk minimizationcited by 73
  15. 2019Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasetscited by 55
  16. 2021Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensemblescited by 54
  17. 2020Distributional Generalization: A New Kind of Generalizationcited by 41
  18. 2021Near-Optimal Linear Regression under Distribution Shiftcited by 40
  19. 2022The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regressioncited by 40
  20. 2020What causes the test error? Going beyond bias-variance via ANOVAcited by 28
  21. 2022The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shiftcited by 25
  22. 2018Distributed linear regression by averagingcited by 21
  23. 2019A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learningcited by 20
  24. 2022A Note on "Assessing Generalization of SGD via Disagreement"cited by 16
  25. 2019WONDER: Weighted one-shot distributed ridge regression in high dimensionscited by 14
  26. 2021On the interplay between data structure and loss function in classification problemscited by 12
  27. 2021When Deep Classifiers Agree: Analyzing Correlations between Learning Order and Image Statisticscited by 11
  28. 2022Task Discovery: Finding the Tasks that Neural Networks Generalize oncited by 9
  29. 2015Applications of Realizations (aka Linearizations) to Free Probabilitycited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 5; 2026: 1.

12 papers.

  1. 2024Strong Model Collapsecited by 62
  2. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 46
  3. 2023Test-Time Adaptation Induces Stronger Accuracy and Agreement-on-the-Linecited by 11
  4. 2025Monitoring Risks in Test-Time Adaptationcited by 10
  5. 2024Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generalisationcited by 9
  6. 2025Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterizationcited by 9
  7. 2025Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structurecited by 4
  8. 2025A Law of Data Reconstruction for Random Features (and Beyond)cited by 3
  9. 2024Signal-Plus-Noise Decomposition of Nonlinear Spiked Random Matrix Modelscited by 2
  10. 2024Implicit Regularization Paths of Weighted Neural Representationscited by 1
  11. 2025auto-fpt: Automating Free Probability Theory Calculations for Machine Learning Theorycited by 0
  12. 2026Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descentcited by 0