# Making sense of AI systems development

arXiv: https://arxiv.org/abs/2408.04311 · HTML: https://mlsanity.com/paper/2408.04311/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Aug 2024.

## Built on by (most cited first, 3 of 3)

- [Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts](https://mlsanity.com/paper/2409.12447/) (2024)
- [Is PMBOK Guide the Right Fit for AI? Re-evaluating Project Management in the Face of Artificial Intelligence Projects](https://mlsanity.com/paper/2506.02214/) (2025)
- [A Configuration-First Framework for Reproducible, Low-Code Machine Learning: a Localization Use Case](https://mlsanity.com/paper/2510.25692/) (2025)

## Abstract (arXiv)

We identify and describe episodes of sensemaking around challenges in modern AI-based systems development that emerged in projects carried out by IBM and client companies. All projects used IBM Watson as the development platform for building tailored AI-based solutions to support workers or customers of the client companies. Yet, many of the projects turned out to be significantly more challenging than IBM and its clients had expected. The analysis reveals that project members struggled to establish reliable meanings about the technology, the project, context, and data to act upon. The project members report multiple aspects of the projects that they were not expecting to need to make sense of yet were problematic. Many issues bear upon the current-generation AI's inherent characteristics, such as dependency on large data sets and continuous improvement as more data becomes available. Those characteristics increase the complexity of the projects and call for balanced mindfulness to avoid unexpected problems.