mlsanityData to 7 Oct 2026

All models are local: time to replace external validation with recurrent local validation

arXiv 2305.03219 ↗First posted 5 May 2023cs.LG stat.MEcited by 2

Not on the map. Why?

Abstract (arXiv)

External validation is often recommended to ensure the generalizability of ML models. However, it neither guarantees generalizability nor equates to a model's clinical usefulness (the ultimate goal of any clinical decision-support tool). External validation is misaligned with current healthcare ML needs. First, patient data changes across time, geography, and facilities. These changes create significant volatility in the performance of a single fixed model (especially for deep learning models, which dominate clinical ML). Second, newer ML techniques, current market forces, and updated regulatory frameworks are enabling frequent updating and monitoring of individual deployed model instances. We submit that external validation is insufficient to establish ML models' safety or utility. Proposals to fix the external validation paradigm do not go far enough. Continued reliance on it as the ultimate test is likely to lead us astray. We propose the MLOps-inspired paradigm of recurring local validation as an alternative that ensures the validity of models while protecting against performance-disruptive data variability. This paradigm relies on site-specific reliability tests before every deployment, followed by regular and recurrent checks throughout the life cycle of the deployed algorithm. Initial and recurrent reliability tests protect against performance-disruptive distribution shifts, and concept drifts that jeopardize patient safety.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 6 of its 45 references that are in the map. First posted on arXiv 5 May 2023.

Builds on

6 of 45 references.

References · 6

  1. 2019Large Scale Incremental Learningcited by 727
  2. 2018End-to-End Incremental Learningcited by 596
  3. 2018A Survey of Unsupervised Deep Domain Adaptationcited by 361
  4. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 273
  5. 2021Deep Stable Learning for Out-Of-Distribution Generalizationcited by 153
  6. 2021Model-Based Domain Generalizationcited by 103

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.