# Combining Q-Learning and Search with Amortized Value Estimates

arXiv: https://arxiv.org/abs/1912.02807 · HTML: https://mlsanity.com/paper/1912.02807/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2019.

## Built on by (most cited first, 19 of 19)

- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [Muesli: Combining Improvements in Policy Optimization](https://mlsanity.com/paper/2104.06159/) (2021)
- [Scalable Online Planning via Reinforcement Learning Fine-Tuning](https://mlsanity.com/paper/2109.15316/) (2021)
- [Evaluating model-based planning and planner amortization for continuous control](https://mlsanity.com/paper/2110.03363/) (2021)
- [Imitating Graph-Based Planning with Goal-Conditioned Policies](https://mlsanity.com/paper/2303.11166/) (2023)
- [Personalized Algorithmic Recourse with Preference Elicitation](https://mlsanity.com/paper/2205.13743/) (2022)
- [Efficient Offline Policy Optimization with a Learned Model](https://mlsanity.com/paper/2210.05980/) (2022)
- [A Survey Analyzing Generalization in Deep Reinforcement Learning](https://mlsanity.com/paper/2401.02349/) (2024)
- [Graph-based Reinforcement Learning meets Mixed Integer Programs: An application to 3D robot assembly discovery](https://mlsanity.com/paper/2203.04120/) (2022)
- [Physics-Aware Combinatorial Assembly Sequence Planning using Data-free Action Masking](https://mlsanity.com/paper/2408.10162/) (2024)
- [Decision Making in Non-Stationary Environments with Policy-Augmented Search](https://mlsanity.com/paper/2401.03197/) (2024)
- [Planning and Learning Using Adaptive Entropy Tree Search](https://mlsanity.com/paper/2102.06808/) (2021)
- [First-Order Problem Solving through Neural MCTS based Reinforcement Learning](https://mlsanity.com/paper/2101.04167/) (2021)
- [Finding the Time to Think: Learning Planning Budgets in Real-Time RL](https://mlsanity.com/paper/2606.26463/) (2026)
- [Counteractive RL: Rethinking Core Principles for Efficient and Scalable Deep Reinforcement Learning](https://mlsanity.com/paper/2603.15871/) (2026)
- [Assessing Policy, Loss and Planning Combinations in Reinforcement Learning using a New Modular Architecture](https://mlsanity.com/paper/2201.02874/) (2022)
- [Online design of dynamic networks](https://mlsanity.com/paper/2410.08875/) (2024)
- [Improve Value Estimation of Q Function and Reshape Reward with Monte Carlo Tree Search](https://mlsanity.com/paper/2410.11642/) (2024)
- [TabQL: In-Context Q-Learning with Tabular Foundation Models](https://mlsanity.com/paper/2605.18979/) (2026)

## Abstract (arXiv)

We introduce "Search with Amortized Value Estimates" (SAVE), an approach for combining model-free Q-learning with model-based Monte-Carlo Tree Search (MCTS). In SAVE, a learned prior over state-action values is used to guide MCTS, which estimates an improved set of state-action values. The new Q-estimates are then used in combination with real experience to update the prior. This effectively amortizes the value computation performed by MCTS, resulting in a cooperative relationship between model-free learning and model-based search. SAVE can be implemented on top of any Q-learning agent with access to a model, which we demonstrate by incorporating it into agents that perform challenging physical reasoning tasks and Atari. SAVE consistently achieves higher rewards with fewer training steps, and---in contrast to typical model-based search approaches---yields strong performance with very small search budgets. By combining real experience with information computed during search, SAVE demonstrates that it is possible to improve on both the performance of model-free learning and the computational cost of planning.