# Rethinking Streaming Machine Learning Evaluation

arXiv: https://arxiv.org/abs/2205.11473 · HTML: https://mlsanity.com/paper/2205.11473/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 10 references that are in the map.
- First posted on arXiv 23 May 2022.

## Builds on (most cited first, 6 of 6)

- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [BREEDS: Benchmarks for Subpopulation Shift](https://mlsanity.com/paper/2008.04859/) (2020)
- [The CLEAR Benchmark: Continual LEArning on Real-World Imagery](https://mlsanity.com/paper/2201.06289/) (2022)
- [A simple way to make neural networks robust against diverse image corruptions](https://mlsanity.com/paper/2001.06057/) (2020)
- [Deconstructing Distributions: A Pointwise Framework of Learning](https://mlsanity.com/paper/2202.09931/) (2022)

## Built on by (most cited first, 5 of 5)

- [Operationalizing Machine Learning: An Interview Study](https://mlsanity.com/paper/2209.09125/) (2022)
- [Monitoring and Adapting ML Models on Mobile Devices](https://mlsanity.com/paper/2305.07772/) (2023)
- [DataCI: A Platform for Data-Centric AI on Streaming Data](https://mlsanity.com/paper/2306.15538/) (2023)
- [Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities](https://mlsanity.com/paper/2312.03812/) (2023)
- [Performance of externally validated machine learning models based on histopathology images for the diagnosis, classification, prognosis, or treatment outcome prediction in female breast cancer: A systematic review](https://mlsanity.com/paper/2312.06697/) (2023)

## Abstract (arXiv)

While most work on evaluating machine learning (ML) models focuses on computing accuracy on batches of data, tracking accuracy alone in a streaming setting (i.e., unbounded, timestamp-ordered datasets) fails to appropriately identify when models are performing unexpectedly. In this position paper, we discuss how the nature of streaming ML problems introduces new real-world challenges (e.g., delayed arrival of labels) and recommend additional metrics to assess streaming ML performance.