mlsanityData to 7 Oct 2026

TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language Modeling Likewise

arXiv 2310.19019 ↗First posted 29 Oct 2023cs.CL cs.AIcited by 15

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large Language Models (LLMs) exhibit impressive reasoning and data augmentation capabilities in various NLP tasks. However, what about small models? In this work, we propose TeacherLM-7.1B, capable of annotating relevant fundamentals, chain of thought, and common mistakes for most NLP samples, which makes annotation more than just an answer, thus allowing other models to learn "why" instead of just "what". The TeacherLM-7.1B model achieved a zero-shot score of 52.3 on MMLU, surpassing most models with over 100B parameters. Even more remarkable is its data augmentation ability. Based on TeacherLM-7.1B, we augmented 58 NLP datasets and taught various student models with different parameters from OPT and BLOOM series in a multi-task setting. The experimental results indicate that the data augmentation provided by TeacherLM has brought significant benefits. We will release the TeacherLM series of models and augmented datasets as open-source.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 36 of its 44 references that are in the map. First posted on arXiv 29 Oct 2023.

Builds on

36 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
23 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  8. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  9. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  10. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  11. 2022Galactica: A Large Language Model for Sciencecited by 736
  12. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  13. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  14. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  15. 2022Large Language Models Are Reasoning Teacherscited by 318
  16. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  17. 2022Can language models learn from explanations in context?cited by 248
  18. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  19. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  20. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  21. 2021CREAK: A Dataset for Commonsense Reasoning over Entity Knowledgecited by 88
  22. 2021Towards Zero-Label Language Learningcited by 76
  23. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 40

Built on by

Cited by year

Citing papers per year: 2024: 12; 2025: 3.

15 papers.

Sort:

By area:

  1. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  2. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 61
  3. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  4. 2024SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuningcited by 32
  5. 2024Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challengescited by 30
  6. 2024MMREC: LLM Based Multi-Modal Recommender Systemcited by 22
  7. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  8. 2024Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarioscited by 12
  9. 2024CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharingcited by 12
  10. 2024Exploring the Limitations of Large Language Models in Compositional Relation Reasoningcited by 9
  11. 2024Towards Foundation-model-based Multiagent System to Accelerate AI for Social Impactcited by 9
  12. 2024Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignmentcited by 7
3 more
  1. 2024Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillationcited by 5
  2. 2025ExpertSteer: Intervening in LLMs through Expert Knowledgecited by 4
  3. 2025Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillationcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2023PolyLM: An Open Source Polyglot Large Language Model
  2. 2026Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation
  3. 2024Rethinking Data Synthesis: A Teacher Model Training Recipe with Interpretation
  4. 2023Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models
  5. 2022Teaching Small Language Models to Reason
  6. 2025Optimising Language Models for Downstream Tasks: A Post-Training Perspective
  7. 2026Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration
  8. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Models