mlsanityData to 14 Oct 2026

Are Large Language Models Truly Smarter Than Humans?

arXiv 2603.16197 ↗First posted 17 Mar 2026cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Public leaderboards increasingly suggest that large language models (LLMs) surpass human experts on benchmarks spanning academic knowledge, law, and programming. Yet most benchmarks are fully public, their questions widely mirrored across the internet, creating systematic risk that models were trained on the very data used to evaluate them. This paper presents three complementary experiments forming a rigorous multi-method contamination audit of six frontier LLMs: GPT-4o, GPT-4o-mini, DeepSeek-R1, DeepSeek-V3, Llama-3.3-70B, and Qwen3-235B. Experiment 1 applies a lexical contamination detection pipeline to 513 MMLU questions across all 57 subjects, finding an overall contamination rate of 13.8% (18.1% in STEM, up to 66.7% in Philosophy) and estimated performance gains of +0.030 to +0.054 accuracy points by category. Experiment 2 applies a paraphrase and indirect-reference diagnostic to 100 MMLU questions, finding accuracy drops by an average of 7.0 percentage points under indirect reference, rising to 19.8 pp in both Law and Ethics. Experiment 3 applies TS-Guessing behavioral probes to all 513 questions and all six models, finding that 72.5% trigger memorization signals far above chance, with DeepSeek-R1 displaying a distributed memorization signature (76.6% partial reconstruction, 0% verbatim recall) that explains its anomalous Experiment 2 profile. All three experiments converge on the same contamination ranking: STEM > Professional > Social Sciences > Humanities.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 12 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

11 of 12 references.

References · 11

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  7. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 332
  8. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 138
  9. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
  10. 2024MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmarkcited by 41
  11. 2024Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?cited by 32

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLM
  2. 2024Testing and Evaluation of Large Language Models: Correctness, Non-Toxicity, and Fairness
  3. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions
  4. 2023Task Contamination: Language Models May Not Be Few-Shot Anymore
  5. 2023From Text to Source: Results in Detecting Large Language Model-Generated Content
  6. 2026Can LLM Reasoning Be Trusted? A Comparative Study: Using Human Benchmarking on Statistical Tasks
  7. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition
  8. 2024Benchmarking Benchmark Leakage in Large Language Models