mlsanityData to 7 Oct 2026

Textbooks Are All You Need

arXiv 2306.11644 ↗First posted 20 Jun 2023cs.CL cs.AI cs.LGcited by 561

Not on the map. Why?

Cited by 95 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook quality" data from the web (6B tokens) and synthetically generated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays surprising emergent properties compared to phi-1-base, our model before our finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller model with 350M parameters trained with the same pipeline as phi-1 that still achieves 45% on HumanEval.

Key facts. Cited by 561 papers in the map (data through 7 Oct 2026). Builds on 35 of its 41 references that are in the map. First posted on arXiv 20 Jun 2023.

Builds on

35 of 41 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
23 more
  1. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  2. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  3. 2023PaLM 2 Technical Reportcited by 1,154
  4. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  5. 2023StarCoder: may the source be with you!cited by 790
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  7. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  9. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  10. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  11. 2023Scaling Data-Constrained Language Modelscited by 391
  12. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  13. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  14. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  15. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  16. 2023The False Promise of Imitating Proprietary LLMscited by 254
  17. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  18. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  19. 2023SantaCoder: don't reach for the stars!cited by 169
  20. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 124
  21. 2023Differentially Private Synthetic Data via Foundation Model APIs 1: Imagescited by 63
  22. 2023Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasingcited by 31
  23. 2023Selective Pre-training for Private Fine-tuningcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 74; 2024: 254; 2025: 124; 2026: 109.

561 papers.

Sort:

By area:

  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  3. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  4. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  5. 2024Phi-4 Technical Reportcited by 675
  6. 2024A Survey on Large Language Models for Code Generationcited by 646
  7. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  8. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  9. 2023The Falcon Series of Open Language Modelscited by 526
  10. 2024Large Language Models: A Surveycited by 506
  11. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  12. 2023Large Language Models for Information Retrieval: A Surveycited by 331
38 more
  1. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  2. 2023Improving Text Embeddings with Large Language Modelscited by 295
  3. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  4. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  5. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  6. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  7. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  8. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  9. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 246
  10. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  11. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  12. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  13. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  14. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 190
  15. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  16. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  17. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  18. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  19. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  20. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  21. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 153
  22. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  23. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  24. 2025Phi-4-reasoning Technical Reportcited by 146
  25. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  26. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 137
  27. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  28. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  29. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  30. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  31. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  32. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  33. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  34. 2023Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialoguecited by 118
  35. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  36. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 112
  37. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  38. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 107

50 most cited of 561 All 561 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
  2. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories
  3. 2026CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models
  4. 2025Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation
  5. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligence
  6. 2023L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
  8. 2025Cross-Task Benchmarking and Evaluation of General-Purpose and Code-Specific Large Language Models ↗