mlsanityData to 14 Oct 2026

ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding

arXiv 2501.07861 ↗First posted 14 Jan 2025cs.CLcited by 32

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

also in 2 more
Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems for Large Language Models (LLMs) hold promise in knowledge-intensive tasks but face limitations in complex multi-step reasoning. While recent methods have integrated RAG with chain-of-thought reasoning or test-time search using Process Reward Models (PRMs), these approaches encounter challenges such as a lack of explanations, bias in PRM training data, early-step bias in PRM scores, and insufficient post-training optimization of reasoning potential. To address these issues, we propose Retrieval-Augmented Reasoning through Trustworthy Process Rewarding (ReARTeR), a framework that enhances RAG systems' reasoning capabilities through post-training and test-time scaling. At test time, ReARTeR introduces Trustworthy Process Rewarding via a Process Reward Model for accurate scalar scoring and a Process Explanation Model (PEM) for generating natural language explanations, enabling step refinement. During post-training, it utilizes Monte Carlo Tree Search guided by Trustworthy Process Rewarding to collect high-quality step-level preference data, optimized through Iterative Preference Optimization. ReARTeR addresses three core challenges: (1) misalignment between PRM and PEM, tackled through off-policy preference learning; (2) bias in PRM training data, mitigated by balanced annotation methods and stronger annotations for challenging examples; and (3) early-step bias in PRM, resolved through a temporal-difference-based look-ahead search strategy. Experimental results on multi-step reasoning benchmarks demonstrate significant improvements, underscoring ReARTeR's potential to advance the reasoning capabilities of RAG systems.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 51 references that are in the map. First posted on arXiv 14 Jan 2025.

Builds on

36 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks2 Oct
  2. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions20 Dec

Other references · 34

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  8. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  9. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  10. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  11. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  12. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
22 more
  1. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  2. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  4. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  5. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  6. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
  7. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  8. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  9. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  10. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
  11. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  12. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 189
  13. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 166
  14. 2024Marco-o1: Towards Open Reasoning Models for Open-Ended Solutionscited by 144
  15. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 138
  16. 2024ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretabilitycited by 101
  17. 2024SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMscited by 67
  18. 2024Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervisioncited by 52
  19. 2024Progressive Multimodal Reasoning via Active Retrievalcited by 25
  20. 2024Enhancing Mathematical Reasoning in LLMs by Stepwise Correctioncited by 16
  21. 2024OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuningcited by 13
  22. 2024A Survey of Controllable Learning: Methods and Applications in Information Retrievalcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 13.

32 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  3. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  4. 2025A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challengescited by 62
  5. 2025From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agentscited by 51
  6. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 42
  7. 2025Synergizing RAG and Reasoning: A Systematic Reviewcited by 27
  8. 2026Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Trainingcited by 18
  9. 2025ComposeRAG: A Modular and Composable RAG for Corpus-Grounded Multi-Hop Question Answeringcited by 11
  10. 2025AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Playcited by 10
  11. 2025HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented Generationcited by 10
  12. 2025ReZero: Enhancing LLM search ability by trying one-more-timecited by 9
20 more
  1. 2025EviNote-RAG: Enhancing RAG Models via Answer-Supportive Evidence Notescited by 8
  2. 2026DLLM-Searcher: Adapting Diffusion Large Language Model for Search Agentscited by 8
  3. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  4. 2025Uncertainty Quantification for Retrieval-Augmented Reasoningcited by 6
  5. 2026Chain-of-Memory: Lightweight Memory Construction with Dynamic Evolution for LLM Agentscited by 6
  6. 2025Curriculum Guided Reinforcement Learning for Efficient Multi Hop Retrieval Augmented Generationcited by 5
  7. 2026Deep Search with Hierarchical Meta-Cognitive Monitoring Inspired by Cognitive Neurosciencecited by 5
  8. 2025RAGXplain: From Explainable Evaluation to Actionable Guidance of RAG Pipelinescited by 4
  9. 2025Prompting Large Language Models with Partial Knowledge for Answering Questions with Unseen Entitiescited by 4
  10. 2026ProRAG: Process-Supervised Reinforcement Learning for Retrieval-Augmented Generationcited by 4
  11. 2025StepMathAgent: A Step-Wise Agent for Evaluating Mathematical Processes through Tree-of-Errorcited by 3
  12. 2025R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Spacecited by 2
  13. 2026SEEK: Steering LLM Reasoning for RAG via Internal Reasoning Sketchescited by 2
  14. 2026SAAS: Self-Aware Reinforcement Learning for Over-Search Mitigation in Agentic Searchcited by 2
  15. 2025From Static to Dynamic: Adaptive Monte Carlo Search for Mathematical Process Supervisioncited by 1
  16. 2026Can David Beat Goliath? On Multi-Hop Reasoning with Resource-Constrained Agentscited by 1
  17. 2026AutoSearch: Adaptive Search Depth for Efficient Agentic RAG via Reinforcement Learningcited by 1
  18. 2026OThink-SRR1: Search, Refine and Reasoning with Reinforced Learning for Large Language Modelscited by 0
  19. 2026StepGap: A Hybrid NLI-LLM Checker for Step-Level Evidence-Gap Detectionin Multi-Hop Question Answeringcited by 0
  20. 2026TreeSeeker: Tree-Structured Trial, Error, and Return in Deep Searchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From Sufficiency to Reflection: Reinforcement-Guided Thinking Quality in Retrieval-Augmented Reasoning for LLMs
  2. 2026PRO-Step: Step-level Process Reward Optimization for Retrieval-Augmented Generation
  3. 2025LeTS: Learning to Think-and-Search via Process-and-Outcome Reward Hybridization
  4. 2025ClueAnchor: Clue-Anchored Knowledge Reasoning Exploration and Optimization for Retrieval-Augmented Generation
  5. 2025SIRAG: Towards Stable and Interpretable RAG with A Process-Supervised Multi-Agent Framework
  6. 2025Retrieval-Augmented Process Reward Model for Generalizable Mathematical Reasoning
  7. 2025RAG+: Enhancing Retrieval-Augmented Generation with Application-Aware Reasoning
  8. 2024RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement