# Hierarchical Decision Making with Structured Policies: A Principled Design via Inverse Optimization

arXiv: https://arxiv.org/abs/2606.28764 · HTML: https://mlsanity.com/paper/2606.28764/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 51 references that are in the map.
- First posted on arXiv 27 Jun 2026.

## Builds on (most cited first, 27 of 27)

- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [A survey on intrinsic motivation in reinforcement learning](https://mlsanity.com/paper/1908.06976/) (2019)
- [State-wise Safe Reinforcement Learning: A Survey](https://mlsanity.com/paper/2302.03122/) (2023)
- [Inverse Optimization: Theory and Applications](https://mlsanity.com/paper/2109.03920/) (2021)
- [Data-Enabled Predictive Control: In the Shallows of the DeePC](https://mlsanity.com/paper/1811.05890/) (2018)
- [Data-Driven Estimation in Equilibrium Using Inverse Optimization](https://mlsanity.com/paper/1308.3397/) (2013)
- [Intelligent problem-solving as integrated hierarchical reinforcement learning](https://mlsanity.com/paper/2208.08731/) (2022)
- [Practical Reinforcement Learning For MPC: Learning from sparse objectives in under an hour on a real robot](https://mlsanity.com/paper/2003.03200/) (2020)
- [Trajectory Planning for Autonomous Vehicles Using Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2011.04752/) (2020)
- [Robotic Table Wiping via Reinforcement Learning and Whole-body Trajectory Optimization](https://mlsanity.com/paper/2210.10865/) (2022)
- [Learning High-Level Policies for Model Predictive Control](https://mlsanity.com/paper/2007.10284/) (2020)
- [Graph Reinforcement Learning for Network Control via Bi-Level Optimization](https://mlsanity.com/paper/2305.09129/) (2023)
- [Enhancing System-Level Safety in Mixed-Autonomy Platoon via Safe Reinforcement Learning](https://mlsanity.com/paper/2401.11148/) (2024)
- [Stability-informed Bayesian Optimization for MPC Cost Function Learning](https://mlsanity.com/paper/2404.12187/) (2024)
- [Interaction-Aware Trajectory Prediction for Safe Motion Planning in Autonomous Driving: A Transformer-Transfer Learning Approach](https://mlsanity.com/paper/2411.01475/) (2024)
- [Quantile Inverse Optimization: Improving Stability in Inverse Linear Programming](https://mlsanity.com/paper/1908.02376/) (2019)
- [Spatial-Temporal Interplay in Human Mobility: A Hierarchical Reinforcement Learning Approach with Hypergraph Representation](https://mlsanity.com/paper/2312.15717/) (2023)
- [Regret Analysis of Learning-Based MPC with Partially-Unknown Cost Function](https://mlsanity.com/paper/2108.02307/) (2021)
- [Random Features Approximation for Control-Affine Systems](https://mlsanity.com/paper/2406.06514/) (2024)
- [Offline Hierarchical Reinforcement Learning via Inverse Optimization](https://mlsanity.com/paper/2410.07933/) (2024)
- [MPC-Inspired Reinforcement Learning for Verifiable Model-Free Control](https://mlsanity.com/paper/2312.05332/) (2023)
- [Neural Horizon Model Predictive Control -- Increasing Computational Efficiency with Neural Networks](https://mlsanity.com/paper/2408.09781/) (2024)
- [Successive Convexification of Non-Convex Optimal Control Problems with State Constraints](https://mlsanity.com/paper/1701.00558/) (2017)
- [Closed-loop Data-Enabled Predictive Control and its equivalence with Closed-loop Subspace Predictive Control](https://mlsanity.com/paper/2402.14374/) (2024)

## Abstract (arXiv)

Hierarchical decision-making frameworks are pivotal for addressing complex control tasks, enabling agents to decompose intricate problems into manageable subgoals. Despite their promise, existing hierarchical policies face critical limitations: (i) reinforcement learning (RL)-based methods struggle to guarantee strict constraint satisfaction, and (ii) optimal control (OC)-based approaches often rely on myopic and computationally prohibitive formulations. To reconcile these trade-offs, hierarchical RL-OC architectures have emerged as a promising paradigm. However, the formulation of the lower-level optimization within these frameworks remains underexplored, often relying on heuristic or myopic objectives. In this work, we propose a principled framework that systematically integrates upper-level goal abstraction with structured lower-level decision making. We adopt an inverse optimization approach to inform the structure of the lower-level problem from expert demonstrations, ensuring that the objective of the lower-level policy remains aligned with the overall long-term task goal. To validate the approach, our framework is evaluated on distinct decision making tasks: network-based resource allocation and continuous collision avoidance. Empirical results demonstrate that our method consistently outperforms strong baselines based on end-to-end RL, learning-augmented optimal control, and existing hierarchical RL approaches in both efficiency and decision quality.