mlsanityData to 14 Oct 2026

Streaming Looking Ahead with Token-level Self-reward

arXiv 2503.00029 ↗First posted 24 Feb 2025cs.LG cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Autoregressive decoding algorithms that use only past information often cannot guarantee the best performance. Recently, people discovered that looking-ahead algorithms such as Monte Carlo Tree Search (MCTS) with external reward models (RMs) can significantly improve models' output by allowing them to think ahead and leverage future outputs and associated rewards to guide the current generation. Such techniques can help the reinforcement fine-tuning phase by sampling better trajectories and the inference phase by selecting the better output. However, their high computational cost limits their applications, especially in streaming scenarios. To address this issue, we propose equipping the policy model with token-level self-reward modeling (TRM) capability to eliminate the need for external models and extra communication. We name the new architecture as Reward Transformer. In addition, we propose a streaming-looking-ahead (SLA) algorithm to further boost search efficiency with better parallelization. Experiments show that SLA achieves an overall win rate of 79.7% against the baseline greedy decoding algorithm on three general-domain datasets with a frozen policy model while maintaining streaming efficiency. If we combine SLA with reinforcement fine-tuning techniques such as DPO, SLA achieves an overall win rate of 89.4%.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 36 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

31 of 36 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
19 more
  1. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  2. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  3. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  4. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  5. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  6. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  7. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  8. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
  9. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 158
  10. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 116
  11. 2023Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decodingcited by 60
  12. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 45
  13. 2024Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Searchcited by 40
  14. 2024LiteSearch: Efficacious Tree Search for LLMcited by 31
  15. 2024Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learningcited by 30
  16. 2023No Train Still Gain. Unleash Mathematical Reasoning of Large Language Models with Monte Carlo Tree Search Guided by Energy Functioncited by 20
  17. 2024VerMCTS: Synthesizing Multi-Step Programs using a Verifier, a Large Language Model, and Tree Searchcited by 14
  18. 2024Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerationscited by 2
  19. 2022An Efficient Dynamic Sampling Policy For Monte Carlo Tree Searchcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Modelcited by 61
  2. 2025WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollbackcited by 17

Similar, not cited

Close in wording, no citation either way

  1. 2026FastGuide: Accelerating Reward Guidance for Diffusion Large Language Models
  2. 2025Reward-Weighted Sampling: Enhancing Non-Autoregressive Characteristics in Masked Diffusion LLMs
  3. 2026Beyond Autoregressive RTG: Conditioning via Injection Outside Sequential Modeling in Decision Transformer
  4. 2026The Missing Piece in Pre-trained Model Evaluation: Reward-Guided Decoding Unlocks Task-Oriented Behavior Without Parameter Updates
  5. 2026Adaptive Decoding via Test-Time Policy Learning for Self-Improving Generation
  6. 2026Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo
  7. 2026Squeezing More from the Stream : Learning Representation Online for Streaming Reinforcement Learning
  8. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decoding