# Relax: An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale

arXiv: https://arxiv.org/abs/2604.11554 · HTML: https://mlsanity.com/paper/2604.11554/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 20 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 17 of 17)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning](https://mlsanity.com/paper/2505.24298/) (2025)
- [Inference-Time Scaling for Generalist Reward Modeling](https://mlsanity.com/paper/2504.02495/) (2025)
- [Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library](https://mlsanity.com/paper/2506.06122/) (2025)
- [Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers](https://mlsanity.com/paper/2510.11370/) (2025)
- [A Survey of Reasoning with Foundation Models](https://mlsanity.com/paper/2312.11562/) (2023)
- [AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training](https://mlsanity.com/paper/2507.01663/) (2025)
- [ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents](https://mlsanity.com/paper/2603.18815/) (2026)

## Built on by (most cited first, 7 of 7)

- [Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence](https://mlsanity.com/paper/2607.07675/) (2026)
- [Rollplex: Cross-Phase GPU Spatial Sharing for Vision Language Model Post-Training](https://mlsanity.com/paper/2608.14498/) (2026)
- [WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editing](https://mlsanity.com/paper/2609.05171/) (2026)
- [Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.21653/) (2026)
- [WeightBridge: An Efficient Weight Transfer Library for Reinforcement Learning](https://mlsanity.com/paper/2609.25442/) (2026)
- [Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions](https://mlsanity.com/paper/2609.25463/) (2026)
- [Coding Agent Memory Post-training: Unlocking the Memory Potential of Pre-trained File Operations for Long-Horizon Tasks via Reinforcement Learning](https://mlsanity.com/paper/2609.34422/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) post-training has proven effective at unlocking reasoning, self-reflection, and tool-use capabilities in large language models. As models extend to omni-modal inputs and agentic multi-turn workflows, RL training systems face three interdependent challenges: heterogeneous data flows, operational robustness at scale, and the staleness -- throughput tradeoff. We present Relax (Reinforcement Engine Leveraging Agentic X-modality), an open-source RL training engine that addresses these challenges through three co-designed architectural layers. First, an omni-native architecture builds multimodal support into the full stack -- from data preprocessing and modality-aware parallelism to inference generation -- rather than retrofitting it onto a text-centric pipeline. Second, each RL role runs as an independent, fault-isolated service that can be scaled, recovered, and upgraded without global coordination. Third, service-level decoupling enables asynchronous training via the TransferQueue data bus, where a single staleness parameter smoothly interpolates among on-policy, near-on-policy, and fully asynchronous execution. Relax achieves a 1.20× end-to-end speedup over veRL on Qwen3-4B on-policy training. Its fully async mode delivers a 1.76× speedup over colocate on Qwen3-4B and a 2.00× speedup on Qwen3-Omni-30B, while all modes converge to the same reward level. Relax supports R3 (Rollout Routing Replay)~citema2025r3 for MoE models with only 1.9% overhead, compared to 32% degradation in veRL under the same configuration. It further demonstrates stable omni-modal RL convergence on Qwen3-Omni across image, text, and audio, sustaining over 2,000 steps on video without degradation. Relax is available at https://github.com/rednote-ai/Relax.