# Two-Timescale Hierarchical Reinforcement Learning for Resilient Operations

arXiv: https://arxiv.org/abs/2607.23434 · HTML: https://mlsanity.com/paper/2607.23434/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 7 of its 70 references that are in the map.
- First posted on arXiv 26 Jul 2026.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 7 of 7)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [MARLIM: Multi-Agent Reinforcement Learning for Inventory Management](https://mlsanity.com/paper/2308.01649/) (2023)
- [Online Pricing and Allocation with Demand Learning and Fulfillment Cost](https://mlsanity.com/paper/2501.18049/) (2025)
- [Dual-Agent Deep Reinforcement Learning for Dynamic Pricing and Replenishment](https://mlsanity.com/paper/2410.21109/) (2024)
- [Reliability and Effectiveness of Autonomous AI Agents in Supply Chain Management](https://mlsanity.com/paper/2605.17036/) (2026)
- [Dynamic Assortment Selection and Pricing with Censored Preference Feedback](https://mlsanity.com/paper/2504.02324/) (2025)
- [Online Joint Assortment-Inventory Optimization under MNL Choices](https://mlsanity.com/paper/2304.02022/) (2023)

## Abstract (arXiv)

Unexpected shocks recur in global operations, requiring decision rules that adapt as market and operating conditions change. Many operational systems also have hierarchical structures in which long-term and short-term decisions pursue a shared objective. We study how hierarchical reinforcement learning can strengthen resilience by adapting these interdependent rules jointly. We develop a two-timescale hierarchical reinforcement learning framework that adapts long-term and short-term policies at their respective time scales. Because the policies are interdependent, we synchronize their updates and prove, to our knowledge, the first convergence guarantees for coupled two-timescale learning. Over T periods, our policies' average gap from an optimal policy pair is O(T-1/2), improving to O(log T/T) when poor decisions produce clearer profit losses. In a used-car case study, inventory replenishment is the long-term decision and customer-arrival pricing the short-term decision. Relative to the strongest partially adaptive benchmark, the framework increases mean profit by 9.2% under joint demand-supply shocks and by 11.8% under a prolonged shock scenario, while maintaining a more stable profit trajectory over time. Short-term adaptation addresses routine seasonality and one-sided disruptions by responding immediately to changing conditions. Under joint demand-supply shocks, however, it is insufficient alone; long-term adaptation is also needed to create favorable conditions for short-term decisions. Joint adaptation thus yields higher and more stable profits through disruption and recovery. Because many organizations already use hierarchical planning, the framework strengthens operational resilience without altering existing decision structures.