# Feature-Based Aggregation and Deep Reinforcement Learning: A Survey and Some New Implementations

arXiv: https://arxiv.org/abs/1804.04577 · HTML: https://mlsanity.com/paper/1804.04577/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2018.

## Built on by (most cited first, 12 of 12)

- [On The Effect of Auxiliary Tasks on Representation Dynamics](https://mlsanity.com/paper/2102.13089/) (2021)
- [Lessons from AlphaZero for Optimal, Model Predictive, and Adaptive Control](https://mlsanity.com/paper/2108.10315/) (2021)
- [Nearly Minimax Optimal Regret for Learning Infinite-horizon Average-reward MDPs with Linear Function Approximation](https://mlsanity.com/paper/2102.07301/) (2021)
- [Approximate Policy Iteration with Bisimulation Metrics](https://mlsanity.com/paper/2202.02881/) (2022)
- [Online Incident Response Planning under Model Misspecification through Bayesian Learning and Belief Quantization](https://mlsanity.com/paper/2508.14385/) (2025)
- [Sampling from Pre-Images to Learn Heuristic Functions for Classical Planning](https://mlsanity.com/paper/2207.03336/) (2022)
- [A Data-Driven State Aggregation Approach for Dynamic Discrete Choice Models](https://mlsanity.com/paper/2304.04916/) (2023)
- [An Adaptive State Aggregation Algorithm for Markov Decision Processes](https://mlsanity.com/paper/2107.11053/) (2021)
- [A Reinforcement Learning Approach to Domain-Knowledge Inclusion Using Grammar Guided Symbolic Regression](https://mlsanity.com/paper/2202.04367/) (2022)
- [Understanding Inverse Reinforcement Learning under Overparameterization: Non-Asymptotic Analysis and Global Optimality](https://mlsanity.com/paper/2503.17865/) (2025)
- [Reputation Bootstrapping for Composite Services using CP-nets](https://mlsanity.com/paper/2105.15135/) (2021)
- [Generalization Through the Lens of Learning Dynamics](https://mlsanity.com/paper/2212.05377/) (2022)

## Abstract (arXiv)

In this paper we discuss policy iteration methods for approximate solution of a finite-state discounted Markov decision problem, with a focus on feature-based aggregation methods and their connection with deep reinforcement learning schemes. We introduce features of the states of the original problem, and we formulate a smaller "aggregate" Markov decision problem, whose states relate to the features. We discuss properties and possible implementations of this type of aggregation, including a new approach to approximate policy iteration. In this approach the policy improvement operation combines feature-based aggregation with feature construction using deep neural networks or other calculations. We argue that the cost function of a policy may be approximated much more accurately by the nonlinear function of the features provided by aggregation, than by the linear function of the features provided by neural network-based reinforcement learning, thereby potentially leading to more effective policy improvement.