# A Data-Driven State Aggregation Approach for Dynamic Discrete Choice Models

arXiv: https://arxiv.org/abs/2304.04916 · HTML: https://mlsanity.com/paper/2304.04916/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 59 references that are in the map.
- First posted on arXiv 11 Apr 2023.

## Builds on (most cited first, 16 of 16)

- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism](https://mlsanity.com/paper/2103.12021/) (2021)
- [Causal Reasoning from Meta-reinforcement Learning](https://mlsanity.com/paper/1901.08162/) (2019)
- [Causal Imitation Learning with Unobserved Confounders](https://mlsanity.com/paper/2208.06267/) (2022)
- [Ivy: Instrumental Variable Synthesis for Causal Inference](https://mlsanity.com/paper/2004.05316/) (2020)
- [Instance-optimal PAC Algorithms for Contextual Bandits](https://mlsanity.com/paper/2207.02357/) (2022)
- [Feature-Based Aggregation and Deep Reinforcement Learning: A Survey and Some New Implementations](https://mlsanity.com/paper/1804.04577/) (2018)
- [Seeker: Real-Time Interactive Search](https://mlsanity.com/paper/1905.13125/) (2019)
- [Partially Linear Additive Gaussian Graphical Models](https://mlsanity.com/paper/1906.03362/) (2019)
- [Temporal Poisson Square Root Graphical Models](https://mlsanity.com/paper/2005.06462/) (2020)
- [Adaptive Experimental Design and Counterfactual Inference](https://mlsanity.com/paper/2210.14369/) (2022)
- [Stochastic Learning for Sparse Discrete Markov Random Fields with Controlled Gradient Approximation Error](https://mlsanity.com/paper/2005.06083/) (2020)
- [Neural Insights for Digital Marketing Content Design](https://mlsanity.com/paper/2302.01416/) (2023)
- [An Efficient Pseudo-likelihood Method for Sparse Binary Pairwise Markov Network Estimation](https://mlsanity.com/paper/1702.08320/) (2017)

## Built on by (most cited first, 3 of 3)

- [Improving Offline RL by Blending Heuristics](https://mlsanity.com/paper/2306.00321/) (2023)
- [Reinforcement Learning Paycheck Optimization for Multivariate Financial Goals](https://mlsanity.com/paper/2403.06011/) (2024)
- [Homomorphic Mappings for Value-Preserving State Aggregation in Markov Decision Processes](https://mlsanity.com/paper/2510.09965/) (2025)

## Abstract (arXiv)

We study dynamic discrete choice models, where a commonly studied problem involves estimating parameters of agent reward functions (also known as "structural" parameters), using agent behavioral data. Maximum likelihood estimation for such models requires dynamic programming, which is limited by the curse of dimensionality. In this work, we present a novel algorithm that provides a data-driven method for selecting and aggregating states, which lowers the computational and sample complexity of estimation. Our method works in two stages. In the first stage, we use a flexible inverse reinforcement learning approach to estimate agent Q-functions. We use these estimated Q-functions, along with a clustering algorithm, to select a subset of states that are the most pivotal for driving changes in Q-functions. In the second stage, with these selected "aggregated" states, we conduct maximum likelihood estimation using a commonly used nested fixed-point algorithm. The proposed two-stage approach mitigates the curse of dimensionality by reducing the problem dimension. Theoretically, we derive finite-sample bounds on the associated estimation error, which also characterize the trade-off of computational complexity, estimation error, and sample complexity. We demonstrate the empirical performance of the algorithm in two classic dynamic discrete choice estimation applications.