# How Can We Develop Explainable Systems? Insights from a Literature Review and an Interview Study

arXiv: https://arxiv.org/abs/2204.01438 · HTML: https://mlsanity.com/paper/2204.01438/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2022.

## Built on by (most cited first, 5 of 5)

- [Tailoring Requirements Engineering for Responsible AI](https://mlsanity.com/paper/2302.10816/) (2023)
- [Invisible Users: Uncovering End-Users' Requirements for Explainable AI via Explanation Forms and Goals](https://mlsanity.com/paper/2302.06609/) (2023)
- [Transcending XAI Algorithm Boundaries through End-User-Inspired Design](https://mlsanity.com/paper/2208.08739/) (2022)
- [Explainable AI: Definition and attributes of a good explanation for health AI](https://mlsanity.com/paper/2409.15338/) (2024)
- [The explanation dialogues: an expert focus study to understand requirements towards explanations within the GDPR](https://mlsanity.com/paper/2501.05325/) (2025)

## Abstract (arXiv)

Quality aspects such as ethics, fairness, and transparency have been proven to be essential for trustworthy software systems. Explainability has been identified not only as a means to achieve all these three aspects in systems, but also as a way to foster users' sentiments of trust. Despite this, research has only marginally focused on the activities and practices to develop explainable systems. To close this gap, we recommend six core activities and associated practices for the development of explainable systems based on the results of a literature review and an interview study. First, we identified and summarized activities and corresponding practices in the literature. To complement these findings, we conducted interviews with 19 industry professionals who provided recommendations for the development process of explainable systems and reviewed the activities and practices based on their expertise and knowledge. We compared and combined the findings of the interviews and the literature review to recommend the activities and assess their applicability in industry. Our findings demonstrate that the activities and practices are not only feasible, but can also be integrated in different development processes.