# Don't Explain without Verifying Veracity: An Evaluation of Explainable AI with Video Activity Recognition

arXiv: https://arxiv.org/abs/2005.02335 · HTML: https://mlsanity.com/paper/2005.02335/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 May 2020.

## Built on by (most cited first, 11 of 11)

- [Soliciting Human-in-the-Loop User Feedback for Interactive Machine Learning Reduces User Trust and Impressions of Model Accuracy](https://mlsanity.com/paper/2008.12735/) (2020)
- [Explainable Activity Recognition for Smart Home Systems](https://mlsanity.com/paper/2105.09787/) (2021)
- [TV-TREES: Multimodal Entailment Trees for Neuro-Symbolic Video Reasoning](https://mlsanity.com/paper/2402.19467/) (2024)
- [Why is plausibility surprisingly problematic as an XAI criterion?](https://mlsanity.com/paper/2303.17707/) (2023)
- [Improving Human Verification of LLM Reasoning through Interactive Explanation Interfaces](https://mlsanity.com/paper/2510.22922/) (2025)
- [Non-contact Multimodal Indoor Human Monitoring Systems: A Survey](https://mlsanity.com/paper/2312.07601/) (2023)
- [STAA: Spatio-Temporal Attention Attribution for Real-Time Interpreting Transformer-based Video Models](https://mlsanity.com/paper/2411.00630/) (2024)
- [Temporal Relevance Analysis for Video Action Models](https://mlsanity.com/paper/2204.11929/) (2022)
- [A Survey of the Various Methodologies Towards making Artificial Intelligence More Explainable](https://mlsanity.com/paper/2311.02291/) (2023)
- ["Trust Junk" Leads to Unjustified Support for Highly Discriminatory Predictive Models](https://mlsanity.com/paper/2607.14152/) (2026)
- [Human-in-the-Loop User Feedback Affects Perceived Accuracy and Trust, but Task Subjectivity Matters](https://mlsanity.com/paper/2607.17548/) (2026)

## Abstract (arXiv)

Explainable machine learning and artificial intelligence models have been used to justify a model's decision-making process. This added transparency aims to help improve user performance and understanding of the underlying model. However, in practice, explainable systems face many open questions and challenges. Specifically, designers might reduce the complexity of deep learning models in order to provide interpretability. The explanations generated by these simplified models, however, might not accurately justify and be truthful to the model. This can further add confusion to the users as they might not find the explanations meaningful with respect to the model predictions. Understanding how these explanations affect user behavior is an ongoing challenge. In this paper, we explore how explanation veracity affects user performance and agreement in intelligent systems. Through a controlled user study with an explainable activity recognition system, we compare variations in explanation veracity for a video review and querying task. The results suggest that low veracity explanations significantly decrease user performance and agreement compared to both accurate explanations and a system without explanations. These findings demonstrate the importance of accurate and understandable explanations and caution that poor explanations can sometimes be worse than no explanations with respect to their effect on user performance and reliance on an AI system.