# AI Application Operations -- A Socio-Technical Framework for Data-driven Organizations

arXiv: https://arxiv.org/abs/2601.06061 · HTML: https://mlsanity.com/paper/2601.06061/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 43 references that are in the map.
- First posted on arXiv 23 Dec 2025.

## Builds on (most cited first, 16 of 16)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Machine Learning Operations (MLOps): Overview, Definition, and Architecture](https://mlsanity.com/paper/2205.02302/) (2022)
- [On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products](https://mlsanity.com/paper/1610.01256/) (2016)
- [Robust Online Monitoring of Signal Temporal Logic](https://mlsanity.com/paper/1506.08234/) (2015)
- [Data-Centric Artificial Intelligence](https://mlsanity.com/paper/2212.11854/) (2022)
- [Data Readiness Report](https://mlsanity.com/paper/2010.07213/) (2020)
- [Robust Conformal Prediction for STL Runtime Verification under Distribution Shift](https://mlsanity.com/paper/2311.09482/) (2023)
- [Data Readiness Levels](https://mlsanity.com/paper/1705.02245/) (2017)
- ["We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning](https://mlsanity.com/paper/2403.16795/) (2024)
- [Monitoring Machine Learning Systems: A Multivocal Literature Review](https://mlsanity.com/paper/2509.14294/) (2025)
- [Verified Runtime Validation for Partially Observable Hybrid Systems](https://mlsanity.com/paper/1811.06502/) (2018)
- [Exploratory Visual Analysis for Increasing Data Readiness in Artificial Intelligence Projects](https://mlsanity.com/paper/2409.03805/) (2024)

## Abstract (arXiv)

We outline a comprehensive framework for artificial intelligence (AI) Application Operations (AIAppOps), based on real-world experiences from diverse organizations. Data-driven projects pose additional challenges to organizations due to their dependency on data across the development and operations cycles. To aid organizations in dealing with these challenges, we present a framework outlining the main steps and roles involved in going from idea to production for data-driven solutions. The data dependency of these projects entails additional requirements on continuous monitoring and feedback, as deviations can emerge in any process step. Therefore, the framework embeds monitoring not merely as a safeguard, but as a unifying feedback mechanism that drives continuous improvement, compliance, and sustained value realization-anchored in both statistical and formal assurance methods that extend runtime verification concepts from safety-critical AI to organizational operations. The proposed framework is structured across core technical processes and supporting services to guide both new initiatives and maturing AI programs.