mlsanityData to 14 Oct 2026

Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training

arXiv 2503.18929 ↗First posted 24 Mar 2025cs.LGcited by 29

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) is a critical component of large language model (LLM) post-training. However, on-policy algorithms used for post-training are not naturally robust to a diversified content of experience replay buffers, which asynchronous off-policy actors can efficiently populate in parallel to training. We propose efficiently learning on such off-policy data via Trajectory Balance with Asynchrony (TBA), an approach to asynchronous RL for LLMs that leverages the principled off-policy TB objective. On math, preference-tuning, and automated red-teaming tasks, we post-train models ranging from Pythia 410M to Qwen 2.5 7B, finding TBA offers speed and performance boosts over strong baselines like Online DPO and Dr. GRPO. Beyond TBA's performance benefits (high accuracy even as asynchrony grows) and speedups (4× or more), we show its reward- and recency-prioritizing sampling enable further gains as data generation is scaled. Our code is available at https://github.com/bbartoldson/TBA.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 60 of its 86 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

60 of 86 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  2. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 55

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
38 more
  1. 2022Red Teaming Language Models with Language Modelscited by 1,026
  2. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  4. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  6. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  7. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 392
  8. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  9. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  10. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  11. 2021GFlowNet Foundationscited by 290
  12. 2018Distributed Prioritized Experience Replaycited by 269
  13. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  14. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 251
  15. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 221
  16. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
  17. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  18. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  19. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  20. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148
  21. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
  22. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  23. 2022Learning GFlowNets from partial episodes for improved convergence and stabilitycited by 132
  24. 2023Amortizing intractable inference in large language modelscited by 97
  25. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 88
  26. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 82
  27. 2020Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path spacecited by 82
  28. 2020VarGrad: A Low-Variance Gradient Estimator for Variational Inferencecited by 75
  29. 2023Towards Understanding and Improving GFlowNet Trainingcited by 75
  30. 2024Amortizing intractable inference in diffusion models for vision, language, and controlcited by 72
  31. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  32. 2023Robust Scheduling with GFlowNetscited by 69
  33. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agentscited by 68
  34. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  35. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarizationcited by 50
  36. 2023Local Search GFlowNetscited by 48
  37. 2024Improved off-policy training of diffusion samplerscited by 48
  38. 2021Podracer architectures for scalable Reinforcement Learningcited by 38

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 18.

29 papers.

Sort:

By area:

  1. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  2. 2025Self-Evolving Curriculum for LLM Reasoningcited by 112
  3. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Modelscited by 75
  4. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 40
  5. 2025RLEP: Reinforcement Learning with Experience Replay for LLM Reasoningcited by 39
  6. 2025Nudging the Boundaries of LLM Reasoningcited by 26
  7. 2025A Comedy of Estimators: On KL Regularization in RL Training of LLMscited by 18
  8. 2025World Modelling Improves Language Model Agentscited by 14
  9. 2026Efficient RL Training for LLMs with Experience Replaycited by 9
  10. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  11. 2025Steering Generative Models with Experimental Data for Protein Fitness Optimizationcited by 7
  12. 2026Reference-guided Policy Optimization for Molecular Optimization via LLM Reasoningcited by 6
17 more
  1. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Trainingcited by 5
  2. 2026f-Trajectory Balance: A Loss Family for Tuning GFlowNets, Generative Models, and LLMs with Off- and On-Policy Datacited by 4
  3. 2025CoPRIS: Efficient and Stable Reinforcement Learning via Concurrency-Controlled Partial Rollout with Importance Samplingcited by 3
  4. 2026Rooted Absorbed Prefix Trajectory Balance with Submodular Replay for GFlowNet Trainingcited by 3
  5. 2026ProActor: Timing-Aware Reinforcement Learning for Proactive Task Scheduling Agentscited by 3
  6. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 2
  7. 2026Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategiescited by 2
  8. 2025Periodic Asynchrony: An On-Policy Approach for Accelerating LLM Reinforcement Learningcited by 1
  9. 2026Beyond Normalization: Rethinking the Partition Function as a Difficulty Scheduler for RLVRcited by 1
  10. 2026Demystifying Design Choices of Reinforcement Fine-tuning: A Batched Contextual Bandit Learning Perspectivecited by 0
  11. 2026GFlowPO: Generative Flow Network as a Language Model Prompt Optimizercited by 0
  12. 2026Approximation of Log-Partition Function in Policy Mirror Descent Induces Implicit Regularization for LLM Post-Trainingcited by 0
  13. 2026OGER: A Robust Offline-Guided Exploration Reward for Hybrid Reinforcement Learningcited by 0
  14. 2026DISA: Offline Importance Sampling for Distribution-Matching LLM-RLcited by 0
  15. 2026Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysiscited by 0
  16. 2026Universality and Convergence of Generative Flowscited by 0
  17. 2026Mixture of GFlowNetscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information
  2. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training
  3. 2026Asynchronous Is Nearly Free for Evolution Strategies on Long-Horizon Agentic Tasks
  4. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs
  5. 2025Laminar: A Scalable Asynchronous RL Post-Training Framework
  6. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training
  7. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
  8. 2026COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning