mlsanityData to 7 Oct 2026

Regularized DeepIV with Model Selection

arXiv 2403.04236 ↗First posted 7 Mar 2024cs.LG econ.EM math.STcited by 6

Not on the map. Why?

Abstract (arXiv)

In this paper, we study nonparametric estimation of instrumental variable (IV) regressions. While recent advancements in machine learning have introduced flexible methods for IV estimation, they often encounter one or more of the following limitations: (1) restricting the IV regression to be uniquely identified; (2) requiring minimax computation oracle, which is highly unstable in practice; (3) absence of model selection procedure. In this paper, we present the first method and analysis that can avoid all three limitations, while still enabling general function approximation. Specifically, we propose a minimax-oracle-free method called Regularized DeepIV (RDIV) regression that can converge to the least-norm IV solution. Our method consists of two stages: first, we learn the conditional distribution of covariates, and by utilizing the learned distribution, we learn the estimator by minimizing a Tikhonov-regularized loss function. We further show that our method allows model selection procedures that can achieve the oracle rates in the misspecified regime. When extended to an iterative estimator, our method matches the current state-of-the-art convergence rate. Our method is a Tikhonov regularized variant of the popular DeepIV method with a non-parametric MLE first-stage estimator, and our results provide the first rigorous guarantees for this empirically used method, showcasing the importance of regularization which was absent from the original work.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 38 of its 84 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

38 of 84 references.

References · 38

  1. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  2. 2019On Gradient Descent Ascent for Nonconvex-Concave Minimax Problemscited by 266
  3. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  4. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  5. 2018Model Evaluation, Model Selection, and Algorithm Selection in Machine Learningcited by 157
  6. 2019Orthogonal Statistical Learningcited by 150
  7. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 141
  8. 2020Near-Optimal Algorithms for Minimax Optimizationcited by 119
  9. 2019Kernel Instrumental Variable Regressioncited by 118
  10. 2021A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentumcited by 106
  11. 2020Efficient Methods for Structured Nonconvex-Nonconcave Min-Max Optimizationcited by 94
  12. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
26 more
  1. 2020Minimax Estimation of Conditional Moment Modelscited by 83
  2. 2019Deep ReLU network approximation of functions on a manifoldcited by 73
  3. 2019Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recoverycited by 73
  4. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  5. 2020Semiparametric proximal causal inferencecited by 58
  6. 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 47
  7. 2021Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approachcited by 43
  8. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  9. 2021Deep Proxy Causal Learning and its Application to Confounded Bandit Policy Evaluationcited by 39
  10. 2019Conditional Density Estimation with Neural Networks: Best Practices and Benchmarkscited by 37
  11. 2020Non-convex Min-Max Optimization: Applications, Challenges, and Recent Theoretical Advancescited by 37
  12. 2018A confounding bridge approach for double negative control inference on causal effectscited by 36
  13. 2022Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systemscited by 35
  14. 2018Adversarial Generalized Method of Momentscited by 34
  15. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
  16. 2021Randomized Stochastic Variance-Reduced Methods for Multi-Task Stochastic Bilevel Optimizationcited by 32
  17. 2019Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Networkcited by 27
  18. 2020Kernel Methods for Unobserved Confounding: Negative Controls, Proxies, and Instrumentscited by 27
  19. 2020The Variational Method of Momentscited by 23
  20. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  21. 2020Instrumental Variable Regression via Kernel Maximum Moment Losscited by 14
  22. 2023Minimax Instrumental Variable Regression and L2 Convergence Guarantees without Identification or Closednesscited by 13
  23. 2022Strategic Decision-Making in the Presence of Information Asymmetry: Provably Efficient RL with Algorithmic Instrumentscited by 9
  24. 2021Localization, Convexity, and Star Aggregationcited by 7
  25. 2021Learning Causal Models from Conditional Moment Restrictions by Importance Weightingcited by 7
  26. 2021Many Proxy Controlscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 1.

6 papers.

  1. 2024Spectral Representation for Causal Estimation with Hidden Confounderscited by 12
  2. 2024Nonparametric Instrumental Regression via Kernel Methods is Minimax Optimalcited by 7
  3. 2025Demystifying Spectral Feature Learning for Instrumental Variable Regressioncited by 4
  4. 2025Optimality and Adaptivity of Deep Neural Features for Instrumental Variable Regressioncited by 3
  5. 2025Learning Theory for Kernel Bilevel Optimizationcited by 2
  6. 2026Adaptive Estimation and Inference in Conditional Moment Models via the Discrepancy Principlecited by 0