mlsanityData to 14 Oct 2026

When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control

arXiv 2605.26418 ↗First posted 26 May 2026cs.LG cs.AI cs.DCcited by 3

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

A properly calibrated rule-based autoscaler can beat every one of six mainstream deep reinforcement learning (DRL) algorithms on cost across every workload we test - so when, if ever, does DRL actually help? We study this in RLScale-Bench, a reproducible benchmark and evaluation protocol for DRL on adaptive resource control, where an agent allocates compute to a dynamic workload under cost and service-level constraints. We evaluate PPO, DQN, A2C, SAC, TD3, and DDPG under matched architectures, training budgets, and reward functions against a calibrated rule-based baseline across six workload patterns and five seeds (240 runs), instantiate the benchmark on Kubernetes Horizontal Pod Autoscaling, and probe distribution-shift generalization. Three findings challenge common assumptions: (i) the calibrated controller achieves the lowest cost on all six workloads, though it trails the best RL agents on bursty and flash traffic; (ii) discrete-action algorithms outperform continuous-action ones by one to two orders of magnitude in constraint violations due to action-space mismatch; and (iii) no single algorithm dominates across workloads, with rankings shifting by up to four positions. The bottleneck in RL-based resource control is not algorithm selection but baseline calibration, reward engineering, and realistic evaluation protocols.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 14 of its 26 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

14 of 26 references.

References · 14

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2016OpenAI Gymcited by 1,736
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  8. 2017Deep Reinforcement Learning that Matterscited by 754
  9. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 93
  10. 2020FIRM: An Intelligent Fine-Grained Resource Management Framework for SLO-Oriented Microservicescited by 41
  11. 2025KIS-S: A GPU-Aware Kubernetes Inference Simulator with RL-Based Auto-Scalingcited by 9
  12. 2025AMP4EC: Adaptive Model Partitioning Framework for Efficient Deep Learning Inference in Edge Computing Environmentscited by 6
2 more
  1. 2025Adaptive GPU Resource Allocation for Multi-Agent Collaborative Reasoning in Serverless Environmentscited by 4
  2. 2025AAPA: An Archetype-Aware Predictive Autoscaler with Uncertainty Quantification for Serverless Workloads on Kubernetescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026EvalStop: Using World Feedback to Detect and Correct Reward Overoptimization in Multi-Tenant RLHF Platformscited by 1
  2. 2026Winning by Peeking: Unenforced Budgets and Test-Set Selection Inflate Short-Budget AutoML Comparisonscited by 1
  3. 2026Budget Boundary Effects in Test-Time Mathematical Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning
  2. 2024Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning
  3. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning
  4. 2026RoAd-RL: A Unified Library and Benchmark for Robust Adversarial Reinforcement Learning
  5. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  6. 2026SAIR: Cost-Efficient Multi-Stage ML Pipeline Autoscaling via In-Context Reinforcement Learning
  7. 2025Value-Based Deep RL Scales Predictably
  8. 2024Can we hop in general? A discussion of benchmark selection and design using the Hopper environment