# Forget What You Know about LLMs Evaluations -- LLMs are Like a Chameleon

arXiv: https://arxiv.org/abs/2502.07445 · HTML: https://mlsanity.com/paper/2502.07445/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://mlsanity.com/paper/2311.04850/) (2023)
- [Platypus: Quick, Cheap, and Powerful Refinement of LLMs](https://mlsanity.com/paper/2308.07317/) (2023)
- [APOLLO: SGD-like Memory, AdamW-level Performance](https://mlsanity.com/paper/2412.05270/) (2024)
- [Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation](https://mlsanity.com/paper/2309.10677/) (2023)
- [Uncovering Overfitting in Large Language Model Editing](https://mlsanity.com/paper/2410.07819/) (2024)
- [Data Contamination Can Cross Language Barriers](https://mlsanity.com/paper/2406.13236/) (2024)
- [Enterprise Benchmarks for Large Language Model Evaluation](https://mlsanity.com/paper/2410.12857/) (2024)

## Built on by (most cited first, 13 of 13)

- [Evaluating Memory Structure in LLM Agents](https://mlsanity.com/paper/2602.11243/) (2026)
- [A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search](https://mlsanity.com/paper/2507.00004/) (2025)
- [PeerRank: Autonomous LLM Evaluation Through Web-Grounded, Bias-Controlled Peer Review](https://mlsanity.com/paper/2602.02589/) (2026)
- [Evaluating the Unseen Capabilities: How Many Theorems Do LLMs Know?](https://mlsanity.com/paper/2506.02058/) (2025)
- [Is General-Purpose AI Reasoning Sensitive to Data-Induced Cognitive Biases? Dynamic Benchmarking on Typical Software Engineering Dilemmas](https://mlsanity.com/paper/2508.11278/) (2025)
- [Concept-Guided Fine-Tuning: Steering ViTs away from Spurious Correlations to Improve Robustness](https://mlsanity.com/paper/2603.08309/) (2026)
- [RTL-BenchMT: Dynamic Maintenance of RTL Generation Benchmark Through Agent-Assisted Analysis and Revision](https://mlsanity.com/paper/2605.15537/) (2026)
- [Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysis](https://mlsanity.com/paper/2608.15630/) (2026)
- [Beyond Confidence: The Rhythms of Reasoning in Generative Models](https://mlsanity.com/paper/2602.10816/) (2026)
- [Interpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assessment Responses](https://mlsanity.com/paper/2608.17810/) (2026)
- [Prompt-Robust Language Models: Which Training Strategies Work?](https://mlsanity.com/paper/2609.01217/) (2026)
- [From Transformers to Weighted Automata: Towards the Verification of Large Language Models](https://mlsanity.com/paper/2610.04569/) (2026)
- [Do correct answers have stable latent knowledge support?](https://mlsanity.com/paper/2610.15277/) (2026)

## Abstract (arXiv)

Large language models (LLMs) often appear to excel on public benchmarks, but these high scores may mask an overreliance on dataset-specific surface cues rather than true language understanding. We introduce the Chameleon Benchmark Overfit Detector (C-BOD), a meta-evaluation framework that systematically distorts benchmark prompts via a parametric transformation and detects overfitting of LLMs. By rephrasing inputs while preserving their semantic content and labels, C-BOD exposes whether a model's performance is driven by memorized patterns. Evaluated on the MMLU benchmark using 26 leading LLMs, our method reveals an average performance degradation of 2.15% under modest perturbations, with 20 out of 26 models exhibiting statistically significant differences. Notably, models with higher baseline accuracy exhibit larger performance differences under perturbation, and larger LLMs tend to be more sensitive to rephrasings, indicating that both cases may overrely on fixed prompt patterns. In contrast, the Llama family and models with lower baseline accuracy show insignificant degradation, suggesting reduced dependency on superficial cues. Moreover, C-BOD's dataset- and model-agnostic design allows easy integration into training pipelines to promote more robust language understanding. Our findings challenge the community to look beyond leaderboard scores and prioritize resilience and generalization in LLM evaluation.