# Towards Compositional Interpretability for XAI

arXiv: https://arxiv.org/abs/2406.17583 · HTML: https://mlsanity.com/paper/2406.17583/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 173 references that are in the map.
- First posted on arXiv 25 Jun 2024.

## Builds on (most cited first, 30 of 41)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey](https://mlsanity.com/paper/2006.11371/) (2020)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences](https://mlsanity.com/paper/1712.00547/) (2017)
- [Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution](https://mlsanity.com/paper/1801.04016/) (2018)
- [lambeq: An Efficient High-Level Python Library for Quantum NLP](https://mlsanity.com/paper/2110.04236/) (2021)
- [Causal Effects of Linguistic Properties](https://mlsanity.com/paper/2010.12919/) (2020)
- [Category Theory in Machine Learning](https://mlsanity.com/paper/2106.07032/) (2021)
- [Sparse Interventions in Language Models with Differentiable Masking](https://mlsanity.com/paper/2112.06837/) (2021)
- [Position: Categorical Deep Learning is an Algebraic Theory of All Architectures](https://mlsanity.com/paper/2402.15332/) (2024)
- [Causal Learning and Explanation of Deep Neural Networks via Autoencoded Activations](https://mlsanity.com/paper/1802.00541/) (2018)
- [Dear XAI Community, We Need to Talk! Fundamental Misconceptions in Current XAI Research](https://mlsanity.com/paper/2306.04292/) (2023)
- [Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomena](https://mlsanity.com/paper/2206.05487/) (2022)
- [The role of causality in explainable artificial intelligence](https://mlsanity.com/paper/2309.09901/) (2023)
- [A CCG-Based Version of the DisCoCat Framework](https://mlsanity.com/paper/2105.07720/) (2021)
- [Causal models in string diagrams](https://mlsanity.com/paper/2304.07638/) (2023)
- [Compositionality as we see it, everywhere around us](https://mlsanity.com/paper/2110.05327/) (2021)
- [An Introduction to String Diagrams for Computer Scientists](https://mlsanity.com/paper/2305.08768/) (2023)
- [Distilling Text into Circuits](https://mlsanity.com/paper/2301.10595/) (2023)
- [Categorical Foundations of Explainable AI: A Unifying Theory](https://mlsanity.com/paper/2304.14094/) (2023)
- [Learning from What's Right and Learning from What's Wrong](https://mlsanity.com/paper/2112.14045/) (2021)
- [Cyber Kittens, or Some First Steps Towards Categorical Cybernetics](https://mlsanity.com/paper/2101.10483/) (2021)
- [A Categorical Semantics of Fuzzy Concepts in Conceptual Spaces](https://mlsanity.com/paper/2110.05985/) (2021)
- [Open Diagrams via Coend Calculus](https://mlsanity.com/paper/2004.04526/) (2020)

## Built on by (most cited first, 12 of 12)

- [Finding Manifolds With Bilinear Autoencoders](https://mlsanity.com/paper/2510.16820/) (2025)
- [Actionable Interpretability Must Be Defined in Terms of Symmetries](https://mlsanity.com/paper/2601.12913/) (2026)
- [DLBacktrace: A Model Agnostic Explainability for any Deep Learning Models](https://mlsanity.com/paper/2411.12643/) (2024)
- [The Standard Interpretable Model: A general theory of interpretable machine learning to deductively design interpretable methods using Lagrangian mechanics](https://mlsanity.com/paper/2606.12289/) (2026)
- [Conversable Complexity: Agentic LLM Collectives as Interpretable Substrates](https://mlsanity.com/paper/2607.01047/) (2026)
- [Quantum Models with Multi-Stage Training for Compositional Concept Generalization](https://mlsanity.com/paper/2608.15601/) (2026)
- [Let the Quantum Creep In: Designing Quantum Neural Network Models by Gradually Swapping Out Classical Components](https://mlsanity.com/paper/2409.17583/) (2024)
- [A Diagrammatic Approach to Improve Computational Efficiency in Group Equivariant Neural Networks](https://mlsanity.com/paper/2412.10837/) (2024)
- [Efficient Generation of Parameterised Quantum Circuits from Large Texts](https://mlsanity.com/paper/2505.13208/) (2025)
- [Interpretability in Deep Time Series Models Demands Semantic Alignment](https://mlsanity.com/paper/2602.02239/) (2026)
- [When AI meets quantum information: A comprehensive review](https://mlsanity.com/paper/2607.00365/) (2026)
- [Graphical Design of Interpretable Architectures](https://mlsanity.com/paper/2608.18936/) (2026)

## Abstract (arXiv)

Artificial intelligence (AI) is currently based largely on black-box machine learning models which lack interpretability. The field of eXplainable AI (XAI) strives to address this major concern, being critical in high-stakes areas such as the finance, legal and health sectors. We present an approach to defining AI models and their interpretability based on category theory. For this we employ the notion of a compositional model, which sees a model in terms of formal string diagrams which capture its abstract structure together with its concrete implementation. This comprehensive view incorporates deterministic, probabilistic and quantum models. We compare a wide range of AI models as compositional models, including linear and rule-based models, (recurrent) neural networks, transformers, VAEs, and causal and DisCoCirc models. Next we give a definition of interpretation of a model in terms of its compositional structure, demonstrating how to analyse the interpretability of a model, and using this to clarify common themes in XAI. We find that what makes the standard 'intrinsically interpretable' models so transparent is brought out most clearly diagrammatically. This leads us to the more general notion of compositionally-interpretable (CI) models, which additionally include, for instance, causal, conceptual space, and DisCoCirc models. We next demonstrate the explainability benefits of CI models. Firstly, their compositional structure may allow the computation of other quantities of interest, and may facilitate inference from the model to the modelled phenomenon by matching its structure. Secondly, they allow for diagrammatic explanations for their behaviour, based on influence constraints, diagram surgery and rewrite explanations. Finally, we discuss many future directions for the approach, raising the question of how to learn such meaningfully structured models in practice.