# The Rollout Infrastructure Tax in Coding-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2607.01415 · HTML: https://mlsanity.com/paper/2607.01415/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 3 of its 16 references that are in the map.
- First posted on arXiv 1 Jul 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 3 of 3)

- [SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent](https://mlsanity.com/paper/2511.16108/) (2025)
- [RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training](https://mlsanity.com/paper/2509.21009/) (2025)
- [RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem](https://mlsanity.com/paper/2011.12719/) (2020)

## Built on by (most cited first, 1 of 1)

- [CUA-Sandbox: Efficient Environments for Computer-Use Agent Reinforcement Learning](https://mlsanity.com/paper/2609.32750/) (2026)

## Abstract (arXiv)

Coding-agent reinforcement learning treats execution infrastructure as a background implementation detail, despite relying on large numbers of interactive software rollouts. This is a missed opportunity: measuring infrastructure overhead can reveal practical efficiency gains for RL post-training, where small per-rollout savings compound at scale. We present a comparative study of four execution substrates: single containers, hosted sandboxes, Kubernetes-orchestrated containers, and cloud virtual machines. We find up to 110× variation in cold-start latency and a 1.8× spread in projected worker-hours for one million 150-step trajectories. Our results suggest that future coding-agent RL systems should optimize execution substrates as part of the training system itself, not merely as deployment plumbing.