# A moving target in AI-assisted decision-making: Dataset shift, model updating, and the problem of update opacity

arXiv: https://arxiv.org/abs/2504.05210 · HTML: https://mlsanity.com/paper/2504.05210/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 58 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Builds on (most cited first, 11 of 11)

- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Manipulating and Measuring Model Interpretability](https://mlsanity.com/paper/1802.07810/) (2018)
- [Contrastive Explanation: A Structural-Model Approach](https://mlsanity.com/paper/1811.03163/) (2018)
- [How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations](https://mlsanity.com/paper/2101.08758/) (2021)
- [Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebooks](https://mlsanity.com/paper/2102.12592/) (2021)
- [The Data Addition Dilemma](https://mlsanity.com/paper/2408.04154/) (2024)
- [Limits of trust in medical AI](https://mlsanity.com/paper/2503.16692/) (2025)

## Built on by (most cited first, 3 of 3)

- [Federated learning, ethics, and the double black box problem in medical AI](https://mlsanity.com/paper/2504.20656/) (2025)
- [Update Opacity: Epistemic Accessibility and Governance Under AI System Change](https://mlsanity.com/paper/2606.00037/) (2026)
- [An empirical evaluation of the risks of AI model updates using clinical data: stability, arbitrariness, and fairness](https://mlsanity.com/paper/2604.23954/) (2026)

## Abstract (arXiv)

Machine learning (ML) systems are vulnerable to performance decline over time due to dataset shift. To address this problem, experts often suggest that ML systems should be regularly updated to ensure ongoing performance stability. Some scholarly literature has begun to address the epistemic and ethical challenges associated with different updating methodologies. Thus far, however, little attention has been paid to the impact of model updating on the ML-assisted decision-making process itself, particularly in the AI ethics and AI epistemology literatures. This article aims to address this gap in the literature. It argues that model updating introduces a new sub-type of opacity into ML-assisted decision-making -- update opacity -- that occurs when users cannot understand how or why an update has changed the reasoning or behaviour of an ML system. This type of opacity presents a variety of distinctive epistemic and safety concerns that available solutions to the black box problem in ML are largely ill-equipped to address. A variety of alternative strategies may be developed or pursued to address the problem of update opacity more directly, including bi-factual explanations, dynamic model reporting, and update compatibility. However, each of these strategies presents its own risks or carries significant limitations. Further research will be needed to address the epistemic and safety concerns associated with model updating and update opacity going forward.