mlsanityData to 14 Oct 2026

ModelDiff: A Framework for Comparing Learning Algorithms

arXiv 2211.12491 ↗First posted 22 Nov 2022cs.LG cs.CV stat.MLcited by 30

Not on the map. Why?

Abstract (arXiv)

We study the problem of (learning) algorithm comparison, where the goal is to find differences between models trained with two different learning algorithms. We begin by formalizing this goal as one of finding distinguishing feature transformations, i.e., input transformations that change the predictions of models trained with one learning algorithm but not the other. We then present ModelDiff, a method that leverages the datamodels framework (Ilyas et al., 2022) to compare learning algorithms based on how they use their training data. We demonstrate ModelDiff through three case studies, comparing models trained with/without data augmentation, with/without pre-training, and with different SGD hyperparameters. Our code is available at https://github.com/MadryLab/modeldiff .

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 56 of its 80 references that are in the map. First posted on arXiv 22 Nov 2022.

Builds on

56 of 80 references.

References · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  8. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  9. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  11. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  12. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
38 more
  1. 2018Sanity Checks for Saliency Mapscited by 1,156
  2. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  3. 2018Do Better ImageNet Models Transfer Better?cited by 565
  4. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  5. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 496
  6. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 476
  7. 2020What is being transferred in transfer learning?cited by 368
  8. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 343
  9. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  10. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  11. 2017Real Time Image Saliency for Black Box Classifierscited by 246
  12. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 245
  13. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
  14. 2021Revisiting Model Stitching to Compare Neural Representationscited by 209
  15. 2022Datamodels: Predicting Predictions from Training Datacited by 189
  16. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  17. 2022Domino: Discovering Systematic Errors with Cross-Modal Embeddingscited by 156
  18. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  19. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 134
  20. 2021Assessing Generalization of SGD via Disagreementcited by 117
  21. 2021Salient ImageNet: How to discover spurious features in Deep Learning?cited by 114
  22. 2018On the importance of single directions for generalizationcited by 99
  23. 2021Learning to See by Looking at Noisecited by 92
  24. 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 87
  25. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  26. 2021A Study of Face Obfuscation in ImageNetcited by 83
  27. 2021Leveraging Sparse Linear Layers for Debuggable Deep Networkscited by 82
  28. 2022Distilling Model Failures as Directions in Latent Spacecited by 82
  29. 2021Similarity and Matching of Neural Network Representationscited by 78
  30. 2022The Effects of Regularization and Data Augmentation are Class Dependentcited by 67
  31. 2020Similarity Analysis of Contextual Word Representation Modelscited by 54
  32. 2021Meaningfully Debugging Model Mistakes using Conceptual Counterfactual Explanationscited by 53
  33. 2021Do Input Gradients Highlight Discriminative Features?cited by 52
  34. 2019Model Similarity Mitigates Test Set Overusecited by 38
  35. 2022Are Vision Transformers Robust to Spurious Correlations?cited by 36
  36. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
  37. 2021Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Levelcited by 34
  38. 20213DB: A Framework for Debugging Computer Vision Modelscited by 31

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 4; 2025: 11; 2026: 3.

30 papers.

Sort:

By area:

  1. 2023TRAK: Attributing Model Behavior at Scalecited by 283
  2. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 131
  3. 2023Describing Differences in Image Sets with Natural Languagecited by 53
  4. 2023The Journey, Not the Destination: How Data Guides Diffusion Modelscited by 43
  5. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 34
  6. 2023FACTS: First Amplify Correlations and Then Slice to Discover Biascited by 26
  7. 2025CUPID: Curating Data your Robot Loves with Influence Functionscited by 23
  8. 2024Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Modelscited by 22
  9. 2025MAGIC: Near-Optimal Data Attribution for Deep Learningcited by 22
  10. 2023Understanding the Detrimental Class-level Effects of Data Augmentationcited by 13
  11. 2023DiG-IN: Diffusion Guidance for Investigating Networks -- Uncovering Classifier Differences Neuron Visualisations and Visual Counterfactual Explanationscited by 11
  12. 2025Simple Mechanistic Explanations for Out-Of-Context Reasoningcited by 10
18 more
  1. 2025AI Supply Chains: An Emerging Ecosystem of AI Actors, Products, and Servicescited by 7
  2. 2023Towards Measuring Representational Similarity of Large Language Modelscited by 6
  3. 2023A Simple and Efficient Baseline for Data Attribution on Imagescited by 5
  4. 2024Data Debiasing with Datamodels (D3M): Improving Subgroup Robustness via Data Selectioncited by 5
  5. 2025Representational Similarity via Interpretable Visual Conceptscited by 5
  6. 2024A Lightweight Measure of Classification Difficulty from Application Dataset Characteristicscited by 4
  7. 2024MD tree: a model-diagnostic tree grown on loss landscapecited by 3
  8. 2025Scaling laws for activation steering with Llama 2 models and refusal mechanismscited by 3
  9. 2026Bergson: An Open Source Library for Data Attributioncited by 3
  10. 2023Investigating the Effects of Fairness Interventions Using Pointwise Representational Similaritycited by 2
  11. 2025Small-to-Large Generalization: Data Influences Models Consistently Across Scalecited by 1
  12. 2025Origin Tracer: A Method for Detecting LoRA Fine-Tuning Origins in LLMscited by 1
  13. 2023Understanding the Benefits of Image Augmentationscited by 0
  14. 2023Perturbed examples reveal invariances shared by language modelscited by 0
  15. 2025ImageSet2Text: Describing Sets of Images through Textcited by 0
  16. 2025RedunCut: Measurement-Driven Sampling and Accuracy Performance Modeling for Low-Cost Live Video Analyticscited by 0
  17. 2026Efficient Neural Network Model Selection for Few-Class Application Datasetscited by 0
  18. 2026Reference-Based Bias Detection in LLMs via Relative Representations of Hidden Statescited by 0