# Variance Reduction for Reinforcement Learning in Input-Driven Environments

arXiv: https://arxiv.org/abs/1807.02264 · HTML: https://mlsanity.com/paper/1807.02264/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jul 2018.

## Built on by (most cited first, 30 of 30)

- [The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models](https://mlsanity.com/paper/2108.06084/) (2021)
- [Is High Variance Unavoidable in RL? A Case Study in Continuous Control](https://mlsanity.com/paper/2110.11222/) (2021)
- [Hindsight Learning for MDPs with Exogenous Inputs](https://mlsanity.com/paper/2207.06272/) (2022)
- [The Role of Baselines in Policy Gradient Optimization](https://mlsanity.com/paper/2301.06276/) (2023)
- [Reinforcement Learning for Branch-and-Bound Optimisation using Retrospective Trajectories](https://mlsanity.com/paper/2205.14345/) (2022)
- [Reinforced Attention for Few-Shot Learning and Beyond](https://mlsanity.com/paper/2104.04192/) (2021)
- [Risk-Aware Transfer in Reinforcement Learning using Successor Features](https://mlsanity.com/paper/2105.14127/) (2021)
- [Reinforcement Learning with History-Dependent Dynamic Contexts](https://mlsanity.com/paper/2302.02061/) (2023)
- [A Survey on Deep Reinforcement Learning for Data Processing and Analytics](https://mlsanity.com/paper/2108.04526/) (2021)
- [CausalSim: A Causal Framework for Unbiased Trace-Driven Simulation](https://mlsanity.com/paper/2201.01811/) (2022)
- [Stochastic Gradient Succeeds for Bandits](https://mlsanity.com/paper/2402.17235/) (2024)
- [Hindsight Value Function for Variance Reduction in Stochastic Dynamic Environment](https://mlsanity.com/paper/2107.12216/) (2021)
- [Toward Robust Long Range Policy Transfer](https://mlsanity.com/paper/2103.02957/) (2021)
- [Online Reinforcement Learning in Non-Stationary Context-Driven Environments](https://mlsanity.com/paper/2302.02182/) (2023)
- [Towards Generalizable Reinforcement Learning for Trade Execution](https://mlsanity.com/paper/2307.11685/) (2023)
- [Variance Reduction for Policy-Gradient Methods via Empirical Variance Minimization](https://mlsanity.com/paper/2206.06827/) (2022)
- [Reinforcement Learning with Exogenous States and Rewards](https://mlsanity.com/paper/2303.12957/) (2023)
- [Exploiting Exogenous Structure for Sample-Efficient Reinforcement Learning](https://mlsanity.com/paper/2409.14557/) (2024)
- [RLGS: Reinforcement Learning-Based Adaptive Hyperparameter Tuning for Gaussian Splatting](https://mlsanity.com/paper/2508.04078/) (2025)
- [Interpretable and Effective Reinforcement Learning for Attacking against Graph-based Rumor Detection](https://mlsanity.com/paper/2201.05819/) (2022)
- [TRAM: Test-Time Risk Adaptation with Mixture of Agents](https://mlsanity.com/paper/2408.08812/) (2024)
- [CNN with large memory layers](https://mlsanity.com/paper/2101.11685/) (2021)
- [Partitioning Distributed Compute Jobs with Reinforcement Learning and Graph Neural Networks](https://mlsanity.com/paper/2301.13799/) (2023)
- [Plume: A Framework for High Performance Deep RL Network Controllers via Prioritized Trace Sampling](https://mlsanity.com/paper/2302.12403/) (2023)
- [CrystalBox: Future-Based Explanations for Input-Driven Deep RL Systems](https://mlsanity.com/paper/2302.13483/) (2023)
- [Reinforcement Learning for Unsupervised Video Summarization with Reward Generator Training](https://mlsanity.com/paper/2407.04258/) (2024)
- [Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed](https://mlsanity.com/paper/2412.10381/) (2024)
- [Cells on Autopilot: Adaptive Cell (Re)Selection via Reinforcement Learning](https://mlsanity.com/paper/2601.04083/) (2026)
- [Is Pure Exploitation Sufficient in Exogenous MDPs with Linear Function Approximation?](https://mlsanity.com/paper/2601.20694/) (2026)
- [Towards Near-Real-Time Telemetry-Aware Routing with Neural Routing Algorithms](https://mlsanity.com/paper/2604.02927/) (2026)

## Abstract (arXiv)

We consider reinforcement learning in input-driven environments, where an exogenous, stochastic input process affects the dynamics of the system. Input processes arise in many applications, including queuing systems, robotics control with disturbances, and object tracking. Since the state dynamics and rewards depend on the input process, the state alone provides limited information for the expected future returns. Therefore, policy gradient methods with standard state-dependent baselines suffer high variance during training. We derive a bias-free, input-dependent baseline to reduce this variance, and analytically show its benefits over state-dependent baselines. We then propose a meta-learning approach to overcome the complexity of learning a baseline that depends on a long sequence of inputs. Our experimental results show that across environments from queuing systems, computer networks, and MuJoCo robotic locomotion, input-dependent baselines consistently improve training stability and result in better eventual policies.