# Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answers

arXiv: https://arxiv.org/abs/2410.11672 · HTML: https://mlsanity.com/paper/2410.11672/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 60 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 30 of 39)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought](https://mlsanity.com/paper/2210.01240/) (2022)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [Lessons from the Trenches on Reproducible Evaluation of Language Models](https://mlsanity.com/paper/2405.14782/) (2024)
- [When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards](https://mlsanity.com/paper/2402.01781/) (2024)
- [An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models](https://mlsanity.com/paper/2007.06778/) (2020)
- [CommonsenseQA 2.0: Exposing the Limits of AI through Gamification](https://mlsanity.com/paper/2201.05320/) (2022)
- [CLadder: Assessing Causal Reasoning in Language Models](https://mlsanity.com/paper/2312.04350/) (2023)
- [Competency Problems: On Finding and Removing Artifacts in Language Data](https://mlsanity.com/paper/2104.08646/) (2021)
- [Shortcut Learning of Large Language Models in Natural Language Understanding](https://mlsanity.com/paper/2208.11857/) (2022)
- [Evaluating Cognitive Maps and Planning in Large Language Models with CogEval](https://mlsanity.com/paper/2309.15129/) (2023)
- [The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants](https://mlsanity.com/paper/1708.01425/) (2017)
- [Auxiliary task demands mask the capabilities of smaller language models](https://mlsanity.com/paper/2404.02418/) (2024)
- [Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets](https://mlsanity.com/paper/1911.09241/) (2019)
- [When Choosing Plausible Alternatives, Clever Hans can be Clever](https://mlsanity.com/paper/1911.00225/) (2019)
- [Benchmarks for Automated Commonsense Reasoning: A Survey](https://mlsanity.com/paper/2302.04752/) (2023)
- [Data Contamination Through the Lens of Time](https://mlsanity.com/paper/2310.10628/) (2023)
- [Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models](https://mlsanity.com/paper/2310.00836/) (2023)
- [Training on the Test Task Confounds Evaluation and Emergence](https://mlsanity.com/paper/2407.07890/) (2024)
- [What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?](https://mlsanity.com/paper/1910.12391/) (2019)

## Built on by (most cited first, 9 of 9)

- [Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation](https://mlsanity.com/paper/2502.06559/) (2025)
- [When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation](https://mlsanity.com/paper/2602.16763/) (2026)
- [Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark](https://mlsanity.com/paper/2504.16137/) (2025)
- [Re-evaluating Theory of Mind evaluation in large language models](https://mlsanity.com/paper/2502.21098/) (2025)
- [Questionnaire Responses Do not Capture the Safety of AI Agents](https://mlsanity.com/paper/2603.14417/) (2026)
- [Are LLMs Ready for Conflict Monitoring? Empirical Evidence from West Africa](https://mlsanity.com/paper/2605.04177/) (2026)
- [LexFlip: A Dissociation Diagnostic for Legal Meaning Preservation Metrics](https://mlsanity.com/paper/2609.05296/) (2026)
- [One Perturbation Is Not Enough: Identifiability and Blind Baselines for Behavioral AI Evaluation](https://mlsanity.com/paper/2609.06190/) (2026)
- [Judging by the Cover: Cleaning LLM Truthfulness Benchmarks to Avoid Surface-Level Feature Leakage](https://mlsanity.com/paper/2609.13003/) (2026)

## Abstract (arXiv)

The integrity of AI benchmarks is fundamental to accurately assess the capabilities of AI systems. The internal validity of these benchmarks - i.e., making sure they are free from confounding factors - is crucial for ensuring that they are measuring what they are designed to measure. In this paper, we explore a key issue related to internal validity: the possibility that AI systems can solve benchmarks in unintended ways, bypassing the capability being tested. This phenomenon, widely known in human and animal experiments, is often referred to as the 'Clever Hans' effect, where tasks are solved using spurious cues, often involving much simpler processes than those putatively assessed. Previous research suggests that language models can exhibit this behaviour as well. In several older Natural Language Processing (NLP) benchmarks, individual n-grams like "not" have been found to be highly predictive of the correct labels, and supervised NLP models have been shown to exploit these patterns. In this work, we investigate the extent to which simple n-grams extracted from benchmark instances can be combined to predict labels in modern multiple-choice benchmarks designed for LLMs, and whether LLMs might be using such n-gram patterns to solve these benchmarks. We show how simple classifiers trained on these n-grams can achieve high scores on several benchmarks, despite lacking the capabilities being tested. Additionally, we provide evidence that modern LLMs might be using these superficial patterns to solve benchmarks. This suggests that the internal validity of these benchmarks may be compromised and caution should be exercised when interpreting LLM performance results on them.