mlsanityData to 7 Oct 2026

Unmasking Clever Hans Predictors and Assessing What Machines Really Learn

arXiv 1902.10178 ↗First posted 26 Feb 2019cs.AI cs.CV cs.LGcited by 359

Not on the map. Why?

Abstract (arXiv)

Current learning machines have successfully solved hard application problems, reaching high accuracy and displaying seemingly "intelligent" behavior. Here we apply recent techniques for explaining decisions of state-of-the-art learning machines and analyze various tasks from computer vision and arcade games. This showcases a spectrum of problem-solving behaviors ranging from naive and short-sighted, to well-informed and strategic. We observe that standard performance evaluation metrics can be oblivious to distinguishing these diverse problem solving behaviors. Furthermore, we propose our semi-automated Spectral Relevance Analysis that provides a practically effective way of characterizing and validating the behavior of nonlinear learning machines. This helps to assess whether a learned model indeed delivers reliably for the problem that it was conceived for. Furthermore, our work intends to add a voice of caution to the ongoing excitement about machine intelligence and pledges to evaluate and judge some of these recent successes in a more nuanced manner.

Key facts. Cited by 359 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 64; 2022: 61; 2023: 66; 2024: 58; 2025: 58; 2026: 52.

359 papers.

Sort:

By area:

  1. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
  2. 2022Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyondcited by 171
  3. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  4. 2022From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagationcited by 141
  5. 2021Benchmarking and Survey of Explanation Methods for Black Box Modelscited by 136
  6. 2021Scaling Up Influence Functionscited by 130
  7. 2023Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directionscited by 123
  8. 2021SpookyNet: Learning Force Fields with Electronic Degrees of Freedom and Nonlocal Effectscited by 103
  9. 2023The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domaincited by 97
  10. 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-trainingcited by 95
  11. 2021Survey of Low-Resource Machine Translationcited by 92
  12. 2021A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Conceptscited by 90
38 more
  1. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  2. 2021Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalizationcited by 83
  3. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  4. 2021Explanations in Autonomous Driving: A Surveycited by 80
  5. 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directionscited by 73
  6. 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 70
  7. 2021This Looks Like That... Does it? Shortcomings of Latent Space Prototype Interpretability in Deep Networkscited by 65
  8. 2023On the Foundations of Shortcut Learningcited by 64
  9. 2023Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcutscited by 63
  10. 2021NICE: An Algorithm for Nearest Instance Counterfactual Explanationscited by 62
  11. 2021Counterfactuals and Causability in Explainable Artificial Intelligence: Theory, Algorithms, and Applicationscited by 59
  12. 2022Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlationcited by 59
  13. 2023Continual Learning: Applications and the Road Forwardcited by 59
  14. 2021Software for Dataset-wide XAI: From Local Explanations to Global Insights with Zennit, CoRelAy, and ViRelAycited by 53
  15. 2022Concept-level Debugging of Part-Prototype Networkscited by 52
  16. 2022Leveraging Explanations in Interactive Machine Learning: An Overviewcited by 52
  17. 2022Beyond Explaining: Opportunities and Challenges of XAI-Based Model Improvementcited by 48
  18. 2022Criteria for Classifying Forecasting Methodscited by 48
  19. 2023Multi-dimensional concept discovery (MCD): A unifying framework with completeness guaranteescited by 48
  20. 2022Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioningcited by 45
  21. 2021This looks more like that: Enhancing Self-Explaining Models by Prototypical Relevance Propagationcited by 43
  22. 2022Optimizing Relevance Maps of Vision Transformers Improves Robustnesscited by 41
  23. 2021Why AI is Harder Than We Thinkcited by 40
  24. 2024RudolfV: A Foundation Model by Pathologists for Pathologistscited by 40
  25. 2022A Survey on Explainable Anomaly Detectioncited by 39
  26. 2024The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learningcited by 39
  27. 2023Improving neural network representations using human similarity judgmentscited by 38
  28. 2021Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safetycited by 35
  29. 2023Neuro-Symbolic Continual Learning: Knowledge, Reasoning Shortcuts and Concept Rehearsalcited by 33
  30. 2021Explainable AI: current status and future directionscited by 32
  31. 2023Labeling Neural Representations with Inverse Recognitioncited by 31
  32. 2024MambaLRP: Explaining Selective State Space Sequence Modelscited by 31
  33. 2021Toward Explainable AI for Regression Modelscited by 30
  34. 2025Token Activation Map to Visually Explain Multimodal LLMscited by 30
  35. 2022One-Pixel Shortcut: on the Learning Preference of Deep Neural Networkscited by 29
  36. 2022Diffeomorphic Counterfactuals with Generative Modelscited by 29
  37. 2025Mechanistic understanding and validation of large AI models with SemanticLenscited by 29
  38. 2023What do neural networks learn in image classification? A frequency shortcut perspectivecited by 28

50 most cited of 359 All 359 (JSON)