# A Free Lunch with Influence Functions? Improving Neural Network Estimates with Concepts from Semiparametric Statistics

arXiv: https://arxiv.org/abs/2202.09096 · HTML: https://mlsanity.com/paper/2202.09096/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 104 references that are in the map.
- First posted on arXiv 18 Feb 2022.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Adapting Neural Networks for the Estimation of Treatment Effects](https://mlsanity.com/paper/1906.02120/) (2019)
- [D'ya like DAGs? A Survey on Structure Learning and Causal Discovery](https://mlsanity.com/paper/2103.02582/) (2021)
- [A Survey on Causal Inference](https://mlsanity.com/paper/2002.02770/) (2020)
- [Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions](https://mlsanity.com/paper/2005.06676/) (2020)
- [Nonparametric Estimation of Heterogeneous Treatment Effects: From Theory to Learning Algorithms](https://mlsanity.com/paper/2101.10943/) (2021)
- [Double/Debiased/Neyman Machine Learning of Treatment Effects](https://mlsanity.com/paper/1701.08687/) (2017)
- [CausalML: Python Package for Causal Machine Learning](https://mlsanity.com/paper/2002.11631/) (2020)
- [Learning Individual Causal Effects from Networked Observational Data](https://mlsanity.com/paper/1906.03485/) (2019)
- [Nested Markov Properties for Acyclic Directed Mixed Graphs](https://mlsanity.com/paper/1701.06686/) (2017)
- [The Influence Function of Semiparametric Estimators](https://mlsanity.com/paper/1508.01378/) (2015)
- [Discriminative Jackknife: Quantifying Uncertainty in Deep Learning via Higher-Order Influence Functions](https://mlsanity.com/paper/2007.13481/) (2020)
- [Semiparametric Inference For Causal Effects In Graphical Models With Hidden Variables](https://mlsanity.com/paper/2003.12659/) (2020)
- [Double Generative Adversarial Networks for Conditional Independence Testing](https://mlsanity.com/paper/2006.02615/) (2020)
- [Visually Communicating and Teaching Intuition for Influence Functions](https://mlsanity.com/paper/1810.03260/) (2018)
- [Causal Mosaic: Cause-Effect Inference via Nonlinear ICA and Ensemble Method](https://mlsanity.com/paper/2001.01894/) (2020)
- [Machine learning in policy evaluation: new tools for causal inference](https://mlsanity.com/paper/1903.00402/) (2019)
- [Intact-VAE: Estimating Treatment Effects under Unobserved Confounding](https://mlsanity.com/paper/2101.06662/) (2021)
- [Higher Order Tangent Spaces and Influence Functions](https://mlsanity.com/paper/1502.00812/) (2015)
- [Smooth, identifiable supermodels of discrete DAG models with latent variables](https://mlsanity.com/paper/1511.06813/) (2015)
- [Efficient adjustment sets for population average treatment effect estimation in non-parametric causal graphical models](https://mlsanity.com/paper/1912.00306/) (2019)
- [Tutorial: Deriving The Efficient Influence Curve for Large Models](https://mlsanity.com/paper/1903.01706/) (2019)
- [Influence Functions for Machine Learning: Nonparametric Estimators for Entropies, Divergences and Mutual Informations](https://mlsanity.com/paper/1411.4342/) (2014)
- [A Semiparametric Approach to Interpretable Machine Learning](https://mlsanity.com/paper/2006.04732/) (2020)
- [Evaluating the Robustness of Targeted Maximum Likelihood Estimators via Realistic Simulations in Nutrition Intervention Trials](https://mlsanity.com/paper/2109.14048/) (2021)

## Built on by (most cited first, 4 of 4)

- [Normalizing Flows for Interventional Density Estimation](https://mlsanity.com/paper/2209.06203/) (2022)
- [Partial Counterfactual Identification of Continuous Outcomes with a Curvature Sensitivity Model](https://mlsanity.com/paper/2306.01424/) (2023)
- [Automated Efficient Estimation using Monte Carlo Efficient Influence Functions](https://mlsanity.com/paper/2403.00158/) (2024)
- [A Causal Research Pipeline and Tutorial for Psychologists and Social Scientists](https://mlsanity.com/paper/2206.05175/) (2022)

## Abstract (arXiv)

Parameter estimation in empirical fields is usually undertaken using parametric models, and such models readily facilitate statistical inference. Unfortunately, they are unlikely to be sufficiently flexible to be able to adequately model real-world phenomena, and may yield biased estimates. Conversely, non-parametric approaches are flexible but do not readily facilitate statistical inference and may still exhibit residual bias. We explore the potential for Influence Functions (IFs) to (a) improve initial estimators without needing more data (b) increase model robustness and (c) facilitate statistical inference. We begin with a broad introduction to IFs, and propose a neural network method 'MultiNet', which seeks the diversity of an ensemble using a single architecture. We also introduce variants on the IF update step which we call 'MultiStep', and provide a comprehensive evaluation of different approaches. The improvements are found to be dataset dependent, indicating an interaction between the methods used and nature of the data generating process. Our experiments highlight the need for practitioners to check the consistency of their findings, potentially by undertaking multiple analyses with different combinations of estimators. We also show that it is possible to improve existing neural networks for `free', without needing more data, and without needing to retrain them.