# When Prediction Error Is Not Enough: Evaluating Nuisance-Function Prediction for Causal Estimation

arXiv: https://arxiv.org/abs/2609.00071 · HTML: https://mlsanity.com/paper/2609.00071/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 12 references that are in the map.
- First posted on arXiv 30 Aug 2026.

## Builds on (most cited first, 4 of 4)

- [Machine learning for causal inference: on the use of cross-fit estimators](https://mlsanity.com/paper/2004.10337/) (2020)
- [Hyperparameter Tuning for Causal Inference with Double Machine Learning: A Simulation Study](https://mlsanity.com/paper/2402.04674/) (2024)
- [Challenges in Obtaining Valid Causal Effect Estimates with Machine Learning Algorithms](https://mlsanity.com/paper/1711.07137/) (2017)
- [An Introduction to Double/Debiased Machine Learning](https://mlsanity.com/paper/2504.08324/) (2025)

## Abstract (arXiv)

Prediction error is widely used to evaluate nuisance-function estimators in causal inference, but its relationship with causal estimator performance may differ across performance measures. We studied this question in a partially linear model using Monte Carlo simulations. We compared ordinary least squares (OLS), generalized additive models (GAMs), XGBoost, and Double Machine Learning with XGBoost (DML-XGBoost), evaluating nuisance-function prediction error, bias, RMSE, and 95% confidence interval coverage. We also examined a simple joint-error measure based on the absolute cross-product of estimation errors from the exposure and outcome nuisance functions. Across the simulated settings, XGBoost had the lowest RMSE among the non-oracle methods, while DML-XGBoost generally provided better confidence interval coverage. Prediction error did not consistently track causal bias across methods and settings, and the method with the best point-estimation performance did not necessarily have the best confidence interval coverage. The joint-error measure was only weakly associated with causal bias and did not provide a useful standalone measure of causal performance. These results suggest that prediction error is useful for assessing nuisance-function estimation, but it should not be treated as a direct measure of the quality of the resulting causal estimator.