# Value-Preserving Action Quantization

arXiv: https://arxiv.org/abs/2610.17252 · HTML: https://mlsanity.com/paper/2610.17252/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 12 Oct 2026.

## Placed in

- Offline RL and imitation learning from data / [Language-conditioned policies and VLAs](https://mlsanity.com/offlinerl/language-conditioned-policies-and-vlas/)

## Builds on (most cited first, 23 of 23)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Monte Carlo Tree Search: A Review of Recent Modifications and Applications](https://mlsanity.com/paper/2103.04931/) (2021)
- [Behavior Generation with Latent Actions](https://mlsanity.com/paper/2403.03181/) (2024)
- [The Surprising Effectiveness of Representation Learning for Visual Imitation](https://mlsanity.com/paper/2112.01511/) (2021)
- [Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions](https://mlsanity.com/paper/2309.10150/) (2023)
- [Action Branching Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1711.08946/) (2017)
- [Learning and Planning in Complex Action Spaces](https://mlsanity.com/paper/2104.06303/) (2021)
- [Discretizing Continuous Action Space for On-Policy Optimization](https://mlsanity.com/paper/1901.10500/) (2019)
- [Monte-Carlo Tree Search as Regularized Policy Optimization](https://mlsanity.com/paper/2007.12509/) (2020)
- [Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning](https://mlsanity.com/paper/2310.11731/) (2023)
- [Solving Continuous Control via Q-learning](https://mlsanity.com/paper/2210.12566/) (2022)
- [Learning to Represent Action Values as a Hypergraph on the Action Vertices](https://mlsanity.com/paper/2010.14680/) (2020)
- [Distributional Policy Optimization: An Alternative Approach for Continuous Control](https://mlsanity.com/paper/1905.09855/) (2019)
- [Continuous Control with Action Quantization from Demonstrations](https://mlsanity.com/paper/2110.10149/) (2021)
- [Discrete Policy: Learning Disentangled Action Space for Multi-Task Robotic Manipulation](https://mlsanity.com/paper/2409.18707/) (2024)
- [Monte Carlo Tree Search with Spectral Expansion for Planning with Dynamical Systems](https://mlsanity.com/paper/2412.11270/) (2024)
- [Model Predictive Trees: Sample-Efficient Receding Horizon Planning with Reusable Tree Search](https://mlsanity.com/paper/2411.15651/) (2024)
- [Spectral Decomposition of Inverse Dynamics for Fast Exploration in Model-Based Manipulation](https://mlsanity.com/paper/2603.27796/) (2026)
- [Difference-Aware Retrieval Policies for Imitation Learning](https://mlsanity.com/paper/2606.09758/) (2026)

## Abstract (arXiv)

Search-based policy learning, such as Monte Carlo tree search (MCTS), offers the flexibility to trade runtime computation for data efficiency and adaptation to novel environments. For robot learning, however, applying search requires discretizing the continuous action space and associated dynamics, typically through action quantization. While reconstruction-based quantization is sufficient for representing the continuous action space alone, policy learning additionally requires the quantized actions to preserve their effects on future states and values. We show that such requirement is not necessarily captured by action reconstruction error. Instead, we prove that the error in the value function resulting from action quantization is tightly bounded by the reconstruction error of state reachability under the quantized actions. Based on these results, we propose value-preserving action quantization (VAQ), which quantizes actions by preserving state reachability. We further combine VAQ with MCTS (VAQ-MCTS) as a value-preserving, search-based policy learning framework for behavioral cloning. Using a wide range of object pushing tasks as a controlled benchmark, we evaluate VAQ-MCTS against state-of-the-art behavioral cloning methods. VAQ-MCTS matches or exceeds their performance under standard settings, while substantially improving data efficiency and robustness to heterogeneous demonstrations, while enabling zero-shot adaptation to novel environments. Hardware experiments further demonstrate the effectiveness of VAQ-MCTS, completing all 30 real-world object pushing trials with only 10 minutes of demonstration data.