# DNA-SE: Towards Deep Neural-Nets Assisted Semiparametric Estimation

arXiv: https://arxiv.org/abs/2408.02045 · HTML: https://mlsanity.com/paper/2408.02045/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 77 references that are in the map.
- First posted on arXiv 4 Aug 2024.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Generative Language Modeling for Automated Theorem Proving](https://mlsanity.com/paper/2009.03393/) (2020)
- [The Marginal Value of Adaptive Gradient Methods in Machine Learning](https://mlsanity.com/paper/1705.08292/) (2017)
- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning](https://mlsanity.com/paper/2010.05627/) (2020)
- [Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1709.01953/) (2017)
- [How Well Generative Adversarial Networks Learn Distributions](https://mlsanity.com/paper/1811.03179/) (2018)
- [Semiparametric proximal causal inference](https://mlsanity.com/paper/2011.08411/) (2020)
- [Superpolynomial Lower Bounds for Learning One-Layer Neural Networks using Gradient Descent](https://mlsanity.com/paper/2006.12011/) (2020)
- [Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives](https://mlsanity.com/paper/1708.09165/) (2017)
- [Machine Learning For Elliptic PDEs: Fast Rate Generalization Bound, Neural Scaling Law and Minimax Optimality](https://mlsanity.com/paper/2110.06897/) (2021)
- [Nonparametric Masked Language Modeling](https://mlsanity.com/paper/2212.01349/) (2022)
- [Subspace Identification for Multi-Source Domain Adaptation](https://mlsanity.com/paper/2310.04723/) (2023)
- [Domain Adaptation as a Problem of Inference on Graphical Models](https://mlsanity.com/paper/2002.03278/) (2020)
- [Quantile regression with deep ReLU Networks: Estimators and minimax rates](https://mlsanity.com/paper/2010.08236/) (2020)
- [Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Models](https://mlsanity.com/paper/2210.16433/) (2022)
- [A Generalized Neyman-Pearson Criterion for Optimal Domain Adaptation](https://mlsanity.com/paper/1810.01545/) (2018)
- [A consensus-based global optimization method for high dimensional machine learning problems](https://mlsanity.com/paper/1909.09249/) (2019)
- [Double Generative Adversarial Networks for Conditional Independence Testing](https://mlsanity.com/paper/2006.02615/) (2020)
- [Gradient Descent for One-Hidden-Layer Neural Networks: Polynomial Convergence and SQ Lower Bounds](https://mlsanity.com/paper/1805.02677/) (2018)
- [Neural Integro-Differential Equations](https://mlsanity.com/paper/2206.14282/) (2022)
- [Deep Learning Methods for Proximal Inference via Maximum Moment Restriction](https://mlsanity.com/paper/2205.09824/) (2022)
- [DeepMed: Semiparametric Causal Mediation Analysis with Debiased Deep Learning](https://mlsanity.com/paper/2210.04389/) (2022)
- [Simplifying debiased inference via automatic differentiation and probabilistic programming](https://mlsanity.com/paper/2405.08675/) (2024)
- [ELSA: Efficient Label Shift Adaptation through the Lens of Semiparametric Models](https://mlsanity.com/paper/2305.19123/) (2023)
- [Ananke: A Python Package For Causal Inference Using Graphical Models](https://mlsanity.com/paper/2301.11477/) (2023)
- [Causal Inference of General Treatment Effects using Neural Networks with A Diverging Number of Confounders](https://mlsanity.com/paper/2009.07055/) (2020)

## Abstract (arXiv)

Semiparametric statistics play a pivotal role in a wide range of domains, including but not limited to missing data, causal inference, and transfer learning, to name a few. In many settings, semiparametric theory leads to (nearly) statistically optimal procedures that yet involve numerically solving Fredholm integral equations of the second kind. Traditional numerical methods, such as polynomial or spline approximations, are difficult to scale to multi-dimensional problems. Alternatively, statisticians may choose to approximate the original integral equations by ones with closed-form solutions, resulting in computationally more efficient, but statistically suboptimal or even incorrect procedures. To bridge this gap, we propose a novel framework by formulating the semiparametric estimation problem as a bi-level optimization problem; and then we develop a scalable algorithm called Deep Neural-Nets Assisted Semiparametric Estimation (DNA-SE) by leveraging the universal approximation property of Deep Neural-Nets (DNN) to streamline semiparametric procedures. Through extensive numerical experiments and a real data analysis, we demonstrate the numerical and statistical advantages of dnase over traditional methods. To the best of our knowledge, we are the first to bring DNN into semiparametric statistics as a numerical solver of integral equations in our proposed general framework.