# Scaling up and Stabilizing Differentiable Planning with Implicit Differentiation

arXiv: https://arxiv.org/abs/2210.13542 · HTML: https://mlsanity.com/paper/2210.13542/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 43 references that are in the map.
- First posted on arXiv 24 Oct 2022.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [General E(2)-Equivariant Steerable CNNs](https://mlsanity.com/paper/1911.08251/) (2019)
- [Value Iteration Networks](https://mlsanity.com/paper/1602.02867/) (2016)
- [Multiscale Deep Equilibrium Models](https://mlsanity.com/paper/2006.08656/) (2020)
- [Imagination-Augmented Agents for Deep Reinforcement Learning](https://mlsanity.com/paper/1707.06203/) (2017)
- [Implicit Deep Learning](https://mlsanity.com/paper/1908.06315/) (2019)
- [Value Prediction Network](https://mlsanity.com/paper/1707.03497/) (2017)
- [Monotone operator equilibrium networks](https://mlsanity.com/paper/2006.08591/) (2020)
- [The Value Equivalence Principle for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2011.03506/) (2020)
- [Exploring Model-based Planning with Policy Networks](https://mlsanity.com/paper/1906.08649/) (2019)
- [Temporal Difference Models: Model-Free Deep RL for Model-Based Control](https://mlsanity.com/paper/1802.09081/) (2018)
- [Stabilizing Equilibrium Models by Jacobian Regularization](https://mlsanity.com/paper/2106.14342/) (2021)
- [An investigation of model-free planning](https://mlsanity.com/paper/1901.03559/) (2019)
- [Model-Augmented Actor-Critic: Backpropagating through Paths](https://mlsanity.com/paper/2005.08068/) (2020)
- [QMDP-Net: Deep Learning for Planning under Partial Observability](https://mlsanity.com/paper/1703.06692/) (2017)
- [Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiation](https://mlsanity.com/paper/2106.03273/) (2021)
- [Proper Value Equivalence](https://mlsanity.com/paper/2106.10316/) (2021)
- [Gated Path Planning Networks](https://mlsanity.com/paper/1806.06408/) (2018)
- [Universal Planning Networks](https://mlsanity.com/paper/1804.00645/) (2018)
- [The Differentiable Cross-Entropy Method](https://mlsanity.com/paper/1909.12830/) (2019)
- [Differentiable Spatial Planning using Transformers](https://mlsanity.com/paper/2112.01010/) (2021)
- [Neural Algorithmic Reasoners are Implicit Planners](https://mlsanity.com/paper/2110.05442/) (2021)
- [Generalized Value Iteration Networks: Life Beyond Lattices](https://mlsanity.com/paper/1706.02416/) (2017)
- [Value Iteration Networks on Multiple Levels of Abstraction](https://mlsanity.com/paper/1905.11068/) (2019)

## Built on by (most cited first, 6 of 6)

- [E(2)-Equivariant Graph Planning for Navigation](https://mlsanity.com/paper/2309.13043/) (2023)
- [Equivariant Action Sampling for Reinforcement Learning and Planning](https://mlsanity.com/paper/2412.12237/) (2024)
- [Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning](https://mlsanity.com/paper/2406.08404/) (2024)
- [Learning to Navigate in Mazes with Novel Layouts using Abstract Top-down Maps](https://mlsanity.com/paper/2412.12024/) (2024)
- [Can Euclidean Symmetry be Leveraged in Reinforcement Learning and Planning?](https://mlsanity.com/paper/2307.08226/) (2023)
- [Compact Bellman-Grounded Cognitive Maps for Cost-Aware Navigation](https://mlsanity.com/paper/2609.05104/) (2026)

## Abstract (arXiv)

Differentiable planning promises end-to-end differentiability and adaptivity. However, an issue prevents it from scaling up to larger-scale problems: they need to differentiate through forward iteration layers to compute gradients, which couples forward computation and backpropagation, and needs to balance forward planner performance and computational cost of the backward pass. To alleviate this issue, we propose to differentiate through the Bellman fixed-point equation to decouple forward and backward passes for Value Iteration Network and its variants, which enables constant backward cost (in planning horizon) and flexible forward budget and helps scale up to large tasks. We study the convergence stability, scalability, and efficiency of the proposed implicit version of VIN and its variants and demonstrate their superiorities on a range of planning tasks: 2D navigation, visual navigation, and 2-DOF manipulation in configuration space and workspace.