mlsanityData to 7 Oct 2026

Heddle: A Distributed Orchestration System for Agentic RL Rollout

arXiv 2603.28101 ↗First posted 30 Mar 2026cs.LGcited by 6

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Agentic Reinforcement Learning (RL) enables LLMs to solve complex tasks by alternating between a data-collection rollout phase and a policy training phase. During rollout, the agent generates trajectories, i.e., multi-step interactions between LLMs and external tools. Yet, frequent tool calls induce long-tailed trajectory generation that bottlenecks rollouts. This stems from step-centric designs that ignore trajectory context, triggering three system problems for long-tail trajectory generation: queueing delays, interference overhead, and inflated per-token time. We propose Heddle, a trajectory-centric system to optimize the when, where, and how of agentic rollout execution. Heddle integrates three core mechanisms: trajectory-level scheduling using runtime prediction and progressive priority to minimize cumulative queueing; trajectory-aware placement via presorted dynamic programming and opportunistic migration during idle tool call intervals to minimize interference; and trajectory-adaptive resource manager that dynamically tunes model parallelism to accelerate the per-token time of long-tail trajectories while maintaining high throughput for short trajectories. Evaluations across diverse agentic RL workloads demonstrate that Heddle effectively neutralizes the long-tail bottleneck, achieving up to 2.5× higher end-to-end rollout throughput compared to state-of-the-art baselines.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 50 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

30 of 50 references.

References · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
18 more
  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  2. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  3. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  5. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  8. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  9. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  10. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  11. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  12. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
  13. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 57
  14. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 56
  15. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 41
  16. 2026CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generationcited by 41
  17. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 21
  18. 2025ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Frontier: Towards Comprehensive and Accurate LLM Inference Simulationcited by 4
  2. 2026ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learningcited by 2
  3. 2026Libra: Efficient Resource Management for Agentic RL Post-Trainingcited by 2
  4. 2026PipeSwift: Revisiting Pipeline Parallelism for Large-Scale Completion-Oriented Agentic LLM Servingcited by 0
  5. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0
  6. 2026QwenGyre: An Elastic Reinforcement Learning Framework for Training xLong-Horizon Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning
  2. 2026EASy: Towards Efficient LLM-Based Agentic System
  3. 2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scale
  4. 2026SyncPlan: Long-Horizon LLM Coordination with Explicit Synchronization and Adaptive Correction
  5. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
  6. 2026Process Reward Informed Tree Rollout for Effective Multi-Turn RL
  7. 2026TIGPO: Temporal Instance-Graph Policy Optimization for Long-Horizon LLM Agents
  8. 2026StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction