mlsanityData to 7 Oct 2026

Distilling LLMs' Decomposition Abilities into Compact Language Models

arXiv 2402.01812 ↗First posted 2 Feb 2024cs.CL cs.AI cs.LGcited by 3

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated proficiency in their reasoning abilities, yet their large size presents scalability challenges and limits any further customization. In contrast, compact models offer customized training but often fall short in solving complex reasoning tasks. This study focuses on distilling the LLMs' decomposition skills into compact models using offline reinforcement learning. We leverage the advancements in the LLM`s capabilities to provide feedback and generate a specialized task-specific dataset for training compact models. The development of an AI-generated dataset and the establishment of baselines constitute the primary contributions of our work, underscoring the potential of compact models in replicating complex problem-solving skills.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 52 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

44 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
31 more
  1. 2023Mistral 7Bcited by 3,065
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  9. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  10. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  11. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  12. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  13. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  14. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  15. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  16. 2023Faithful Chain-of-Thought Reasoningcited by 285
  17. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  18. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 155
  19. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  20. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  21. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  22. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  23. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 76
  24. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  25. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 68
  26. 2023Anti-Exploration by Random Network Distillationcited by 49
  27. 2022Automatic Generation of Socratic Subquestions for Teaching Math Word Problemscited by 42
  28. 2021UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planningcited by 23
  29. 2022Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flowscited by 21
  30. 2023Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex Reasoningcited by 16
  31. 2023Katakomba: Tools and Benchmarks for Data-Driven NetHackcited by 5

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2024textttLM^texttt2: A Simple Society of Language Models Solves Complex Reasoningcited by 7
  3. 2024NeSy is alive and well: A LLM-driven symbolic approach for better code comment data generation and classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Models
  2. 2025How Difficulty-Aware Staged Reinforcement Learning Enhances LLMs' Reasoning Capabilities: A Preliminary Experimental Study
  3. 2025Bridging the Reasoning Gap: Small LLMs Can Plan with Generalised Strategies
  4. 2025DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Training
  5. 2024Divide-or-Conquer? Which Part Should You Distill Your LLM?
  6. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMs
  7. 2026Training LLMs for Divide-and-Conquer Reasoning Elevates Test-Time Scalability
  8. 2025Distilling LLM Agent into Small Models with Retrieval and Code Tools