mlsanityData to 14 Oct 2026

STEPER: Step-wise Knowledge Distillation for Enhancing Reasoning Ability in Multi-Step Retrieval-Augmented Language Models

arXiv 2510.07923 ↗First posted 9 Oct 2025cs.CL cs.AIcited by 1

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

also in 2 more
Abstract (arXiv)

Answering complex real-world questions requires step-by-step retrieval and integration of relevant information to generate well-grounded responses. However, existing knowledge distillation methods overlook the need for different reasoning abilities at different steps, hindering transfer in multi-step retrieval-augmented frameworks. To address this, we propose Stepwise Knowledge Distillation for Enhancing Reasoning Ability in Multi-Step Retrieval-Augmented Language Models (StepER). StepER employs step-wise supervision to align with evolving information and reasoning demands across stages. Additionally, it incorporates difficulty-aware training to progressively optimize learning by prioritizing suitable steps. Our method is adaptable to various multi-step retrieval-augmented language models, including those that use retrieval queries for reasoning paths or decomposed questions. Extensive experiments show that StepER outperforms prior methods on multi-hop QA benchmarks, with an 8B model achieving performance comparable to a 70B teacher model.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 43 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

23 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  2. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions20 Dec

Other references · 21

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2023A Survey of Large Language Modelscited by 2,576
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  7. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  8. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  9. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  10. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
  11. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 346
  12. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
9 more
  1. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 256
  2. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  3. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 184
  4. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 159
  5. 2024SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMscited by 67
  6. 2024Retrieve, Summarize, Plan: Advancing Multi-hop Question Answering with an Iterative Approachcited by 40
  7. 2023SAIL: Search-Augmented Instruction Learningcited by 33
  8. 2024Mentor-KD: Making Small Language Models Better Multi-step Reasonerscited by 13
  9. 2020A Simple General Approach to Balance Task Difficulty in Multi-Task Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026EGTR-Review: Efficient Evidence-Grounded Scientific Peer Review Generation via Multi-Agent Teacher Distillationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026PRO-Step: Step-level Process Reward Optimization for Retrieval-Augmented Generation
  2. 2025Graph-Augmented Reasoning: Evolving Step-by-Step Knowledge Graph Retrieval for LLM Reasoning
  3. 2025DeepRAG: Thinking to Retrieve Step by Step for Large Language Models
  4. 2023Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks
  5. 2026FutureMind: Equipping Small Language Models with Strategic Thinking-Pattern Priors via Adaptive Knowledge Distillation
  6. 2024ProcBench: Benchmark for Multi-Step Reasoning and Following Procedure
  7. 2025Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoning
  8. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization