mlsanityData to 14 Oct 2026

Using Kolmogorov-Smirnov Distance for Measuring Distribution Shift in Machine Learning

arXiv 2510.15996 ↗First posted 14 Oct 2025cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

One of the major problems in Machine Learning (ML) and Artificial Intelligence (AI) is the fact that the probability distribution of the test data in the real world could deviate substantially from the probability distribution of the training data set. When this happens, the predictions of an ML system or an AI agent could involve large errors which is very troublesome and undesirable. While this is a well-known hard problem plaguing the AI and ML systems' accuracy and reliability, in certain applications such errors could be critical for safety and reliability of AI and ML systems. One approach to deal with this problem is to monitor and measure the deviation in the probability distribution of the test data in real time and to compensate for this deviation. In this paper, we propose and explore the use of Kolmogorov-Smirnov (KS) Test for measuring the distribution shift and we show how the KS distance can be used to quantify the distribution shift and its impact on an AI agent's performance. Our results suggest that KS distance could be used as a valuable statistical tool for monitoring and measuring the distribution shift. More specifically, it is shown that even a distance of KS=0.02 could lead to about 50% increase in the travel time at a single intersection using a Reinforcement Learning agent which is quite significant. It is hoped that the use of KS Test and KS distance in AI-based smart transportation could be an important step forward for gauging the performance degradation of an AI agent in real time and this, in turn, could help the AI agent to cope with the distribution shift in a more informed manner.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 8 of its 22 references that are in the map. First posted on arXiv 14 Oct 2025.

Builds on

8 of 22 references.

References · 8

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  2. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  3. 2016An overview of gradient descent optimization algorithmscited by 1,033
  4. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
  5. 2021A Fine-Grained Analysis on Distribution Shiftcited by 171
  6. 2022From Concept Drift to Model Degradation: An Overview on Performance-Aware Drift Detectorscited by 83
  7. 2014Domain Adaptive Neural Networks for Object Recognitioncited by 52
  8. 2020GeneraLight: Improving Environment Generalization of Traffic Signal Control via Meta Reinforcement Learningcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Self-Adaptive Multi-Agent LLM-Based Security Pattern Selection for IoT Systemscited by 0
  2. 2026When Drift Detectors cry Wolf: False Alarm Rates in continuous ML Monitoringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ShiftKD: Benchmarking Knowledge Distillation under Distribution Shift
  2. 2023Distribution Shift Inversion for Out-of-Distribution Prediction
  3. 2024Assessing the Impact of Distribution Shift on Reinforcement Learning Performance
  4. 2025When to retrain a machine learning model
  5. 2024Prediction Accuracy & Reliability: Classification and Object Localization under Distribution Shift
  6. 2022Distribution inference risks: Identifying and mitigating sources of leakage
  7. 2025Probabilistic Consistency in Machine Learning and Its Connection to Uncertainty Quantification
  8. 2024Between Randomness and Arbitrariness: Some Lessons for Reliable Machine Learning at Scale