# Perturbing the Derivative: Wild Refitting for Model-Free Evaluation of Machine Learning Models under Bregman Losses

arXiv: https://arxiv.org/abs/2509.02476 · HTML: https://mlsanity.com/paper/2509.02476/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 69 references that are in the map.
- First posted on arXiv 2 Sep 2025.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Transformers as Soft Reasoners over Language](https://mlsanity.com/paper/2002.05867/) (2020)
- [PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance](https://mlsanity.com/paper/2306.05443/) (2023)
- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://mlsanity.com/paper/1811.12808/) (2018)
- [Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences](https://mlsanity.com/paper/2404.12272/) (2024)
- [ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modeling](https://mlsanity.com/paper/2405.17743/) (2024)
- [Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1911.06854/) (2019)
- [Stability and Generalization of Learning Algorithms that Converge to Global Optima](https://mlsanity.com/paper/1710.08402/) (2017)
- [A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking](https://mlsanity.com/paper/2302.14301/) (2023)
- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)
- [The gap between theory and practice in function approximation with deep neural networks](https://mlsanity.com/paper/2001.07523/) (2020)
- [Foundations of Reinforcement Learning and Interactive Decision Making](https://mlsanity.com/paper/2312.16730/) (2023)
- [Practical Large-Scale Linear Programming using Primal-Dual Hybrid Gradient](https://mlsanity.com/paper/2106.04756/) (2021)
- [On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domains](https://mlsanity.com/paper/2305.02657/) (2023)
- [Fine Tuning LLM for Enterprise: Practical Guidelines and Recommendations](https://mlsanity.com/paper/2404.10779/) (2024)
- [Generative AI-enabled Vehicular Networks: Fundamentals, Framework, and Case Study](https://mlsanity.com/paper/2304.11098/) (2023)
- [Multi-Armed Bandits With Machine Learning-Generated Surrogate Rewards](https://mlsanity.com/paper/2506.16658/) (2025)
- [Is K-fold cross validation the best model selection method for Machine Learning?](https://mlsanity.com/paper/2401.16407/) (2024)
- [Eigenvalue Decay Implies Polynomial-Time Learnability for Neural Networks](https://mlsanity.com/paper/1708.03708/) (2017)
- [Contextual Online Decision Making with Infinite-Dimensional Functional Regression](https://mlsanity.com/paper/2501.18359/) (2025)
- [Wild refitting for black box prediction](https://mlsanity.com/paper/2506.21460/) (2025)
- [Sample Efficient Reinforcement Learning with Partial Dynamics Knowledge](https://mlsanity.com/paper/2312.12558/) (2023)

## Abstract (arXiv)

We study the excess risk evaluation of classical penalized empirical risk minimization (ERM) with Bregman losses. We show that by leveraging the idea of wild refitting, one can efficiently upper bound the excess risk through the so-called "wild optimism," without relying on the global structure of the underlying function class. This property makes our approach inherently model-free. Unlike conventional analysis, our framework operates with just one dataset and black-box access to the training procedure. The method involves randomized Rademacher symmetrization and constructing artificially modified outputs by perturbation in the derivative space with appropriate scaling, upon which we retrain a second predictor for excess risk estimation. We establish high-probability performance guarantee under the fixed design setting, demonstrating that wild refitting under Bregman losses, with an appropriately chosen wild noise scale, yields a valid upper bound on the excess risk. Thus, our work is promising for theoretically evaluating modern opaque ML models, such as deep neural networks and generative models, where the function class is too complex for classical learning theory and empirical process techniques.