# Perturbing the Derivative: Doubly Wild Refitting for Model-Free Evaluation of Opaque Machine Learning Predictors

arXiv: https://arxiv.org/abs/2511.18789 · HTML: https://mlsanity.com/paper/2511.18789/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 49 references that are in the map.
- First posted on arXiv 24 Nov 2025.

## Builds on (most cited first, 11 of 11)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modeling](https://mlsanity.com/paper/2405.17743/) (2024)
- [GPT Models in Construction Industry: Opportunities, Limitations, and a Use Case Validation](https://mlsanity.com/paper/2305.18997/) (2023)
- [On Dynamic Pricing with Covariates](https://mlsanity.com/paper/2112.13254/) (2021)
- [Is K-fold cross validation the best model selection method for Machine Learning?](https://mlsanity.com/paper/2401.16407/) (2024)
- [Eigenvalue Decay Implies Polynomial-Time Learnability for Neural Networks](https://mlsanity.com/paper/1708.03708/) (2017)
- [Contextual Online Decision Making with Infinite-Dimensional Functional Regression](https://mlsanity.com/paper/2501.18359/) (2025)
- [Wild refitting for black box prediction](https://mlsanity.com/paper/2506.21460/) (2025)
- [ADMM for monotone operators: convergence analysis and rates](https://mlsanity.com/paper/1705.01913/) (2017)

## Built on by (most cited first, 2 of 2)

- [Simultaneous Pointwise Majorization for Mixed Tail Processes with Applications in Gaussian Chaos and Ergodic Diffusions](https://mlsanity.com/paper/2609.01576/) (2026)
- [Sharp Structure-Agnostic Minimax Risk for Partial Linear Models](https://mlsanity.com/paper/2609.07997/) (2026)

## Abstract (arXiv)

We study the problem of excess risk evaluation for empirical risk minimization (ERM) under convex losses. We show that by leveraging the idea of wild refitting, one can upper bound the excess risk through the so-called "wild optimism," without relying on the global structure of the underlying function class but only assuming black box access to the training algorithm and a single dataset. We begin by generating two sets of artificially modified pseudo-outcomes created by stochastically perturbing the derivatives with carefully chosen scaling. Using these pseudo-labeled datasets, we refit the black-box procedure twice to obtain two wild predictors and derive an efficient excess risk upper bound under the fixed design setting. Requiring no prior knowledge of the complexity of the underlying function class, our method is essentially model-free and holds significant promise for theoretically evaluating modern opaque deep neural networks and generative models, where traditional learning theory could be infeasible due to the extreme complexity of the hypothesis class.