Data to 7 Oct 2026

Reinforced Self-Training (ReST) for Language Modeling

arXiv 2308.08998 ↗First posted 17 Aug 2023cs.CL cs.LGcited by 486

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 2 more

Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training
Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) can improve the quality of large language model's (LLM) outputs by aligning them with human preferences. We propose a simple algorithm for aligning LLMs with human preferences inspired by growing batch reinforcement learning (RL), which we call Reinforced Self-Training (ReST). Given an initial LLM policy, ReST produces a dataset by generating samples from the policy, which are then used to improve the LLM policy using offline RL algorithms. ReST is more efficient than typical online RLHF methods because the training dataset is produced offline, which allows data reuse. While ReST is a general approach applicable to all generative learning settings, we focus on its application to machine translation. Our results show that ReST can substantially improve translation quality, as measured by automated metrics and human evaluation on machine translation benchmarks in a compute and sample-efficient manner.

Key facts. Cited by 486 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 69 references that are in the map. First posted on arXiv 17 Aug 2023.

Builds on

43 of 69 references.

Closest ancestors · 4

Same direction, newest first

Other references · 39

2017Attention Is All You Needcited by 59,932
2020Language Models are Few-Shot Learnerscited by 32,199
2022Training language models to follow instructions with human feedbackcited by 15,712
2017Proximal Policy Optimization Algorithmscited by 14,441
2014Sequence to Sequence Learning with Neural Networkscited by 2,974
2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
2020Learning to summarize from human feedbackcited by 2,562
2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
2022Competition-Level Code Generation with AlphaCodecited by 1,427
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406

27 more

2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
2020COMET: A Neural Framework for MT Evaluationcited by 1,024
2022Red Teaming Language Models with Language Modelscited by 983
2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
2022Solving math word problems with process- and outcome-based feedbackcited by 871
2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
2022Improving alignment of dialogue agents via targeted human judgementscited by 527
2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
2021Recursively Summarizing Books with Human Feedbackcited by 287
2022Defining and Characterizing Reward Hackingcited by 184
2019Revisiting Self-Training for Neural Sequence Generationcited by 154
2018Self-Imitation Learningcited by 135
2021Scaling Laws for Neural Machine Translationcited by 110
2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 73
2020Text Generation by Learning from Demonstrationscited by 64
2020Countering Language Drift with Seeded Iterated Learningcited by 57
2021Machine Translation Decoding beyond Beam Searchcited by 55
2021Regularized Behavior Value Estimationcited by 32
2023Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasingcited by 31
2021Launchpad: A Programming Model for Distributed Machine Learning Researchcited by 14
2020Supervised Seeded Iterated Learning for Interactive Language Learningcited by 7
2022MAD for Robust Reinforcement Learning in Machine Translationcited by 7
2020Self-Training for Unsupervised Neural Machine Translation in Unbalanced Training Data Scenarioscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 33; 2024: 179; 2025: 136; 2026: 138.

486 papers.

Sort:

By area:

2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
2024Self-Rewarding Language Modelscited by 654
2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325

38 more

2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 294
2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
2024Iterative Reasoning Preference Optimizationcited by 244
2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
2024Many-Shot In-Context Learningcited by 203
2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
2023Mitigating the Alignment Tax of RLHFcited by 181
2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
2024Stream of Search (SoS): Learning to Search in Languagecited by 141
2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 135
2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120

50 most cited of 486 All 486 (JSON)

Similar, not cited

Close in wording, no citation either way