A Data-Driven State Aggregation Approach for Dynamic Discrete Choice Models
Not on the map. Why?
Abstract (arXiv)
We study dynamic discrete choice models, where a commonly studied problem involves estimating parameters of agent reward functions (also known as "structural" parameters), using agent behavioral data. Maximum likelihood estimation for such models requires dynamic programming, which is limited by the curse of dimensionality. In this work, we present a novel algorithm that provides a data-driven method for selecting and aggregating states, which lowers the computational and sample complexity of estimation. Our method works in two stages. In the first stage, we use a flexible inverse reinforcement learning approach to estimate agent Q-functions. We use these estimated Q-functions, along with a clustering algorithm, to select a subset of states that are the most pivotal for driving changes in Q-functions. In the second stage, with these selected "aggregated" states, we conduct maximum likelihood estimation using a commonly used nested fixed-point algorithm. The proposed two-stage approach mitigates the curse of dimensionality by reducing the problem dimension. Theoretically, we derive finite-sample bounds on the associated estimation error, which also characterize the trade-off of computational complexity, estimation error, and sample complexity. We demonstrate the empirical performance of the algorithm in two classic dynamic discrete choice estimation applications.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 16 of its 59 references that are in the map. First posted on arXiv 11 Apr 2023.
Builds on
16 of 59 references.
References · 16
- 2016Generative Adversarial Imitation Learningcited by 1,597
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
- 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
- 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
- 2019Causal Reasoning from Meta-reinforcement Learningcited by 53
- 2022Causal Imitation Learning with Unobserved Confounderscited by 41
- 2020Ivy: Instrumental Variable Synthesis for Causal Inferencecited by 17
- 2022Instance-optimal PAC Algorithms for Contextual Banditscited by 16
- 2018Feature-Based Aggregation and Deep Reinforcement Learning: A Survey and Some New Implementationscited by 12
- 2019Seeker: Real-Time Interactive Searchcited by 9
- 2019Partially Linear Additive Gaussian Graphical Modelscited by 6
- 2020Temporal Poisson Square Root Graphical Modelscited by 6
4 more
- 2022Adaptive Experimental Design and Counterfactual Inferencecited by 5
- 2020Stochastic Learning for Sparse Discrete Markov Random Fields with Controlled Gradient Approximation Errorcited by 4
- 2023Neural Insights for Digital Marketing Content Designcited by 4
- 2017An Efficient Pseudo-likelihood Method for Sparse Binary Pairwise Markov Network Estimationcited by 3
Built on by
Citing papers per year: 2023: 1; 2024: 1; 2025: 1.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees
- 2026A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models
- 2025Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models
- 2024Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions
- 2025An Empirical Risk Minimization Approach for Offline Inverse RL and Dynamic Discrete Choice Model
- 2025Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimation
- 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism
- 2025Recursive Reward Aggregation