mlsanityData to 14 Oct 2026

Towards Interpretability in Audio and Visual Affective Machine Learning: A Review

arXiv 2306.08933 ↗First posted 15 Jun 2023cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Machine learning is frequently used in affective computing, but presents challenges due the opacity of state-of-the-art machine learning methods. Because of the impact affective machine learning systems may have on an individual's life, it is important that models be made transparent to detect and mitigate biased decision making. In this regard, affective machine learning could benefit from the recent advancements in explainable artificial intelligence (XAI) research. We perform a structured literature review to examine the use of interpretability in the context of affective machine learning. We focus on studies using audio, visual, or audiovisual data for model training and identified 29 research articles. Our findings show an emergence of the use of interpretability methods in the last five years. However, their use is currently limited regarding the range of methods used, the depth of evaluations, and the consideration of use-cases. We outline the main gaps in the research and provide recommendations for researchers that aim to implement interpretable methods for affective machine learning.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 38 of its 167 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

38 of 167 references.

References · 38

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  2. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  3. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  4. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,552
  5. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  6. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  7. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  8. 2014How transferable are features in deep neural networks?cited by 1,225
  9. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  10. 2018Sanity Checks for Saliency Mapscited by 1,156
  11. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,089
  12. 2016The Mythos of Model Interpretabilitycited by 721
26 more
  1. 2019Explaining Machine Learning Classifiers through Diverse Counterfactual Explanationscited by 614
  2. 2019A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAIcited by 404
  3. 2022From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagationcited by 141
  4. 2020Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectorscited by 120
  5. 2017Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciencescited by 85
  6. 2022Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanationscited by 85
  7. 2019Measuring the Quality of Explanations: The System Causability Scale (SCS). Comparing Human and Machine Explanationscited by 64
  8. 2022A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advancescited by 59
  9. 2022VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformerscited by 53
  10. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 51
  11. 2020GANterfactual - Counterfactual Explanations for Medical Non-Experts using Generative Adversarial Learningcited by 50
  12. 2021Affective Image Content Analysis: Two Decades Review and New Perspectivescited by 47
  13. 2022MultiViz: Towards Visualizing and Understanding Multimodal Modelscited by 36
  14. 2019Global Aggregations of Local Explanations for Black Box modelscited by 30
  15. 2021M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysiscited by 29
  16. 2021Towards Relatable Explainable AI with the Perceptual Processcited by 27
  17. 2021Deep Learning for Depression Recognition with Audiovisual Cues: A Reviewcited by 24
  18. 2018Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignmentcited by 20
  19. 2022Comparing Explanation Methods for Traditional Machine Learning Models Part 1: An Overview of Current Methods and Quantifying Their Disagreementcited by 14
  20. 2019Characterizing Sources of Uncertainty to Proxy Calibration and Disambiguate Annotator and Data Biascited by 11
  21. 2022Interpretability for Multimodal Emotion Recognition using Concept Activation Vectorscited by 8
  22. 2019Global Saliency: Aggregating Saliency Maps to Assess Dataset Artefact Biascited by 6
  23. 2019Human-Centered Emotion Recognition in Animated GIFscited by 4
  24. 2020Toward Multimodal Modeling of Emotional Expressivenesscited by 3
  25. 2022Assessing Gender Bias in Predictive Algorithms using eXplainable AIcited by 2
  26. 2021Comparing Facial Expression Recognition in Humans and Machines: Using CAM, GradCAM, and Extremal Perturbationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Beyond saliency: enhancing explanation of speech emotion recognition with expert-referenced acoustic cuescited by 0
  2. 2026AffectSeek: Agentic Affective Understanding in Long Videos under Vague User Queriescited by 0
  3. 2026Organization of Valence and Arousal in Vision-Language Representations of Built Environments: Insights from the EMOIS Datasetcited by 0