mlsanityData to 14 Oct 2026

Textbooks Are All You Need II: phi-1.5 technical report

arXiv 2309.05463 ↗First posted 11 Sep 2023cs.CL cs.AIcited by 565

Not on the map. Why?

Cited by 66 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We continue the investigation into the power of smaller Transformer-based language models as initiated by TinyStories -- a 10 million parameter model that can produce coherent English -- and the follow-up work on phi-1, a 1.3 billion parameter model with Python coding performance close to the state-of-the-art. The latter work proposed to use existing Large Language Models (LLMs) to generate ``textbook quality" data as a way to enhance the learning process compared to traditional web data. We follow the ``Textbooks Are All You Need" approach, focusing this time on common sense reasoning in natural language, and create a new 1.3 billion parameter model named phi-1.5, with performance on natural language tasks comparable to models 5x larger, and surpassing most non-frontier LLMs on more complex reasoning tasks such as grade-school mathematics and basic coding. More generally, phi-1.5 exhibits many of the traits of much larger LLMs, both good -- such as the ability to ``think step by step" or perform some rudimentary in-context learning -- and bad, including hallucinations and the potential for toxic and biased generations -- encouragingly though, we are seeing improvement on that front thanks to the absence of web data. We open-source phi-1.5 to promote further research on these urgent topics.

Key facts. Cited by 565 papers in the map (data through 14 Oct 2026). Builds on 28 of its 39 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

28 of 39 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  12. 2021Program Synthesis with Large Language Modelscited by 3,637
16 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  8. 2023Textbooks Are All You Needcited by 569
  9. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  10. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  11. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  12. 2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
  13. 2023An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Modelscited by 24
  14. 2021First Steps of an Approach to the ARC Challenge based on Descriptive Grid Models and the Minimum Description Length Principlecited by 13
  15. 2021Identify, Align, and Integrate: Matching Knowledge Graphs to Commonsense Reasoning Taskscited by 9
  16. 2019Answering questions by learning to rank -- Learning to rank by answering questionscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 41; 2024: 297; 2025: 151; 2026: 76.

565 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  6. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  7. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  8. 2024Phi-4 Technical Reportcited by 711
  9. 2024A Survey on Large Language Models for Code Generationcited by 658
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  11. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  12. 2023The Falcon Series of Open Language Modelscited by 535
38 more
  1. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 533
  2. 2024Large Language Models: A Surveycited by 525
  3. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  5. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  6. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  7. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 281
  8. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  9. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  10. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 236
  11. 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
  12. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  13. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 225
  14. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  15. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  16. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  17. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 174
  18. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  19. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  20. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  21. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  22. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  23. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  24. 2024The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learningcited by 145
  25. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 144
  26. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  27. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 141
  28. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  29. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 129
  30. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  31. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 125
  32. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 124
  33. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 122
  34. 2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 120
  35. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimizationcited by 118
  36. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 116
  37. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  38. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 115

50 most cited of 565 All 565 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic
  2. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  3. 2025Evaluating Large Language Models on the 2026 Korean CSAT Mathematics Exam: Measuring Mathematical Ability in a Zero-Data-Leakage Setting
  4. 2024CLR-Bench: Evaluating Large Language Models in College-level Reasoning
  5. 2025Cross-Task Benchmarking and Evaluation of General-Purpose and Code-Specific Large Language Models ↗
  6. 2026More Accurate or More Efficient? Evaluating Locally Deployed Compact Open-Weight Language Models for Mathematical Reasoning
  7. 2026SLMJury: Can Small Language Models Judge as Well as Large Ones?
  8. 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data