# Sequential Knockoffs for Variable Selection in Reinforcement Learning

arXiv: https://arxiv.org/abs/2303.14281 · HTML: https://mlsanity.com/paper/2303.14281/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 65 references that are in the map.
- First posted on arXiv 24 Mar 2023.

## Builds on (most cited first, 23 of 23)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [Representation Learning for Online and Offline RL in Low-rank MDPs](https://mlsanity.com/paper/2110.04652/) (2021)
- [Controlling the false discovery rate via knockoffs](https://mlsanity.com/paper/1404.5609/) (2014)
- [Batch Policy Learning in Average Reward Markov Decision Processes](https://mlsanity.com/paper/2007.11771/) (2020)
- [Estimating Dynamic Treatment Regimes in Mobile Health Using V-learning](https://mlsanity.com/paper/1611.03531/) (2016)
- [Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settings](https://mlsanity.com/paper/2001.04515/) (2020)
- [Learning Markov State Abstractions for Deep Reinforcement Learning](https://mlsanity.com/paper/2106.04379/) (2021)
- [Deep Knockoffs](https://mlsanity.com/paper/1811.06687/) (2018)
- [On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluation](https://mlsanity.com/paper/2201.06169/) (2022)
- [Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Making](https://mlsanity.com/paper/2002.01751/) (2020)
- [Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learning](https://mlsanity.com/paper/2110.10719/) (2021)
- [Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient](https://mlsanity.com/paper/2011.04019/) (2020)
- [RANK: Large-Scale Inference with Graphical Nonlinear Knockoffs](https://mlsanity.com/paper/1709.00092/) (2017)
- [Sample Efficient Feature Selection for Factored MDPs](https://mlsanity.com/paper/1703.03454/) (2017)
- [High-Dimensional Knockoffs Inference for Time Series Data](https://mlsanity.com/paper/2112.09851/) (2021)
- [Sufficient Markov Decision Processes with Alternating Deep Neural Networks](https://mlsanity.com/paper/1704.07531/) (2017)
- [Sequential Advantage Selection for Optimal Treatment Regimes](https://mlsanity.com/paper/1405.5239/) (2014)
- [Relaxing the Assumptions of Knockoffs by Conditioning](https://mlsanity.com/paper/1903.02806/) (2019)
- [Variable Selection in Regression-based Estimation of Dynamic Treatment Regimes](https://mlsanity.com/paper/2101.07359/) (2021)

## Built on by (most cited first, 7 of 7)

- [Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic](https://mlsanity.com/paper/2603.01162/) (2026)
- [Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions](https://mlsanity.com/paper/2601.15353/) (2026)
- [Off-policy Evaluation with Deeply-abstracted States](https://mlsanity.com/paper/2406.19531/) (2024)
- [TERC: A Transfer Entropy Redundancy Criterion for State Variable Selection in Reinforcement Learning](https://mlsanity.com/paper/2401.11512/) (2024)
- [To Switch or Not to Switch? Balanced Policy Switching in Offline Reinforcement Learning](https://mlsanity.com/paper/2407.01837/) (2024)
- [Where to Intervene: Action Selection in Deep Reinforcement Learning](https://mlsanity.com/paper/2507.04187/) (2025)
- [Generalized Fitted Q-Iteration with Clustered Data](https://mlsanity.com/paper/2510.03912/) (2025)

## Abstract (arXiv)

In real-world applications of reinforcement learning, it is often challenging to obtain a state representation that is parsimonious and satisfies the Markov property without prior knowledge. Consequently, it is common practice to construct a state larger than necessary, e.g., by concatenating measurements over contiguous time points. However, needlessly increasing the dimension of the state may slow learning and obfuscate the learned policy. We introduce the notion of a minimal sufficient state in a Markov decision process (MDP) as the subvector of the original state under which the process remains an MDP and shares the same reward function as the original process. We propose a novel SEquEntial Knockoffs (SEEK) algorithm that estimates the minimal sufficient state in a system with high-dimensional complex nonlinear dynamics. In large samples, the proposed method achieves selection consistency. As the method is agnostic to the reinforcement learning algorithm being applied, it benefits downstream tasks such as policy learning. Empirical experiments verify theoretical results and show the proposed approach outperforms several competing methods regarding variable selection accuracy and regret.