mlsanityData to 14 Oct 2026

Pretraining on the Test Set Is All You Need

arXiv 2309.08632 ↗First posted 13 Sep 2023cs.CL cs.AIcited by 33

Not on the map. Why?

Abstract (arXiv)

Inspired by recent work demonstrating the promise of smaller Transformer-based language models pretrained on carefully curated data, we supercharge such approaches by investing heavily in curating a novel, high quality, non-synthetic data mixture based solely on evaluation benchmarks. Using our novel dataset mixture consisting of less than 100 thousand tokens, we pretrain a 1 million parameter transformer-based LLM phi-CTNL (pronounced ``fictional") that achieves perfect results across diverse academic benchmarks, strictly outperforming all known foundation models. phi-CTNL also beats power-law scaling and exhibits a never-before-seen grokking-like ability to accurately predict downstream evaluation benchmarks' canaries.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 13 Sep 2023.

Builds on

32 of 47 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2021Program Synthesis with Large Language Modelscited by 3,637
20 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  5. 2023PaLM 2 Technical Reportcited by 1,162
  6. 2021Scaling Vision Transformerscited by 999
  7. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  8. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  9. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  10. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  11. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  12. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  13. 2021Scaling Laws for Transfercited by 272
  14. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  15. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  16. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  17. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  18. 2023Grokking modular arithmeticcited by 83
  19. 2021Scaling Scaling Laws with Board Gamescited by 63
  20. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Modelcited by 38

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 14; 2025: 11; 2026: 2.

33 papers.

Sort:

By area:

  1. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  2. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
  3. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 138
  4. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 117
  5. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 101
  6. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 79
  7. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 43
  8. 2024When Large Language Models Meet Vector Databases: A Surveycited by 39
  9. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language modelscited by 35
  10. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 33
  11. 2024Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediationcited by 30
  12. 2025Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcutscited by 29
21 more
  1. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 19
  2. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competitioncited by 18
  3. 2025Efficient Prediction of Pass@k Scaling in Large Language Modelscited by 16
  4. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  5. 2024Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Modelscited by 15
  6. 2024Benchmark Inflation: Revealing LLM Performance Gaps Using Retro-Holdoutscited by 14
  7. 2025Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMscited by 11
  8. 2023Leveraging Word Guessing Games to Assess the Intelligence of Large Language Modelscited by 8
  9. 2024PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Modelscited by 8
  10. 2024BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Modelscited by 7
  11. 2025Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarkscited by 7
  12. 2024EnviroExam: Benchmarking Environmental Science Knowledge of Large Language Modelscited by 5
  13. 2025IberBench: LLM Evaluation on Iberian Languagescited by 5
  14. 2023Post Turing: Mapping the landscape of LLM Evaluationcited by 4
  15. 2024Balancing Speciality and Versatility: A Coarse to Fine Framework for Mitigating Catastrophic Forgetting in Large Language Modelscited by 3
  16. 2025Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarkscited by 3
  17. 2025Evaluating the Robustness of Chinchilla Compute-Optimal Scalingcited by 2
  18. 2026Quantifying the Effect of Test Set Contamination on Generative Evaluationscited by 2
  19. 2025Understand User Opinions of Large Language Models via LLM-Powered In-the-Moment User Experience Interviewscited by 1
  20. 2024Kotlin ML Pack: Technical Reportcited by 0
  21. 2025Finetune Once: Decoupling General & Domain Learning with Dynamic Boosted Annealingcited by 0