# Inference on Optimal Dynamic Policies via Softmax Approximation

arXiv: https://arxiv.org/abs/2303.04416 · HTML: https://mlsanity.com/paper/2303.04416/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 98 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 28 of 28)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Bridging the Gap Between Value and Policy Based Reinforcement Learning](https://mlsanity.com/paper/1702.08892/) (2017)
- [Equivalence Between Policy Gradients and Soft Q-Learning](https://mlsanity.com/paper/1704.06440/) (2017)
- [Orthogonal Statistical Learning](https://mlsanity.com/paper/1901.09036/) (2019)
- [Confidence Intervals and Hypothesis Testing for High-Dimensional Regression](https://mlsanity.com/paper/1306.3171/) (2013)
- [Approximate Residual Balancing: De-Biased Inference of Average Treatment Effects in High Dimensions](https://mlsanity.com/paper/1604.07125/) (2016)
- [De-biasing the Lasso: Optimal Sample Size for Gaussian Designs](https://mlsanity.com/paper/1508.02757/) (2015)
- [Adversarial Estimation of Riesz Representers](https://mlsanity.com/paper/2101.00009/) (2020)
- [De-Biased Machine Learning of Global and Local Parameters Using Regularized Riesz Representers](https://mlsanity.com/paper/1802.08667/) (2018)
- [Augmented Minimax Linear Estimation](https://mlsanity.com/paper/1712.00038/) (2017)
- [Program Evaluation and Causal Inference with High-Dimensional Data](https://mlsanity.com/paper/1311.2645/) (2013)
- [Double/Debiased Machine Learning for Dynamic Treatment Effects via g-Estimation](https://mlsanity.com/paper/2002.07285/) (2020)
- [A General Theory of Hypothesis Tests and Confidence Regions for Sparse High Dimensional Models](https://mlsanity.com/paper/1412.8765/) (2014)
- [Statistically Efficient Off-Policy Policy Gradients](https://mlsanity.com/paper/2002.04014/) (2020)
- [Hypothesis Testing in High-Dimensional Regression under the Gaussian Random Design Model: Asymptotic Theory](https://mlsanity.com/paper/1301.4240/) (2013)
- [Off-policy Confidence Sequences](https://mlsanity.com/paper/2102.09540/) (2021)
- [Asymptotic normality and optimalities in estimation of large Gaussian graphical models](https://mlsanity.com/paper/1309.6024/) (2013)
- [Uniform Post Selection Inference for LAD Regression and Other Z-estimation problems](https://mlsanity.com/paper/1304.0282/) (2013)
- [Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons](https://mlsanity.com/paper/2202.13163/) (2022)
- [Confidence intervals for high-dimensional inverse covariance estimation](https://mlsanity.com/paper/1403.6752/) (2014)
- [Incremental causal effects](https://mlsanity.com/paper/1907.13258/) (2019)
- [A Unified Theory of Confidence Regions and Testing for High Dimensional Estimating Equations](https://mlsanity.com/paper/1510.08986/) (2015)
- [Semi-parametric efficiency bounds for high-dimensional models](https://mlsanity.com/paper/1601.00815/) (2016)
- [Uniform Inference for High-dimensional Quantile Regression: Linear Functionals and Regression Rank Scores](https://mlsanity.com/paper/1702.06209/) (2017)
- [Confidence regions for high-dimensional generalized linear models under sparsity](https://mlsanity.com/paper/1610.01353/) (2016)
- [Breaking the curse of dimensionality in regression](https://mlsanity.com/paper/1708.00430/) (2017)
- [Adaptive Semi-Supervised Inference for Optimal Treatment Decisions with Electronic Medical Record Data](https://mlsanity.com/paper/2203.02318/) (2022)

## Built on by (most cited first, 3 of 3)

- [Policy Learning with Distributional Welfare](https://mlsanity.com/paper/2311.15878/) (2023)
- [Statistical Inference and Learning for Shapley Additive Explanations (SHAP)](https://mlsanity.com/paper/2602.10532/) (2026)
- [Policy Learning with Abstention](https://mlsanity.com/paper/2510.19672/) (2025)

## Abstract (arXiv)

Estimating optimal dynamic policies from offline data is a fundamental problem in dynamic decision making. In the context of causal inference, the problem is known as estimating the optimal dynamic treatment regime. Even though there exists a plethora of methods for estimation, constructing confidence intervals for the value of the optimal regime and structural parameters associated with it is inherently harder, as it involves non-linear and non-differentiable functionals of unknown quantities that need to be estimated. Prior work resorted to sub-sample approaches that can deteriorate the quality of the estimate. We show that a simple soft-max approximation to the optimal treatment regime, for an appropriately fast growing temperature parameter, can achieve valid inference on the truly optimal regime. We illustrate our result for a two-period optimal dynamic regime, though our approach should directly extend to the finite horizon case. Our work combines techniques from semi-parametric inference and g-estimation, together with an appropriate triangular array central limit theorem, as well as a novel analysis of the asymptotic influence and asymptotic bias of softmax approximations.