# Disentangled Concepts Speak Louder Than Words: Explainable Video Action Recognition

arXiv: https://arxiv.org/abs/2511.03725 · HTML: https://mlsanity.com/paper/2511.03725/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 64 references that are in the map.
- First posted on arXiv 5 Nov 2025.

## Builds on (most cited first, 30 of 42)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024)
- [Post-hoc Concept Bottleneck Models](https://mlsanity.com/paper/2205.15480/) (2022)
- [Label-Free Concept Bottleneck Models](https://mlsanity.com/paper/2304.06129/) (2023)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)
- [Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers](https://mlsanity.com/paper/2106.05392/) (2021)
- [CRAFT: Concept Recursive Activation FacTorization for Explainability](https://mlsanity.com/paper/2211.10154/) (2022)
- [Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI](https://mlsanity.com/paper/2010.07487/) (2020)
- [CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks](https://mlsanity.com/paper/2204.10965/) (2022)
- [Efficient Parameter-free Clustering Using First Neighbor Relations](https://mlsanity.com/paper/1902.11266/) (2019)
- [Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2212.04500/) (2022)
- [Leveraging Sparse Linear Layers for Debuggable Deep Networks](https://mlsanity.com/paper/2105.04857/) (2021)
- [Incremental Residual Concept Bottleneck Models](https://mlsanity.com/paper/2404.08978/) (2024)
- [HAA500: Human-Centric Atomic Action Dataset with Curated Videos](https://mlsanity.com/paper/2009.05224/) (2020)
- [Coarse-to-Fine Concept Bottleneck Models](https://mlsanity.com/paper/2310.02116/) (2023)

## Built on by (most cited first, 2 of 2)

- [Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs](https://mlsanity.com/paper/2605.22823/) (2026)
- [InstrAct: Towards Action-Centric Understanding in Instructional Videos](https://mlsanity.com/paper/2604.08762/) (2026)

## Abstract (arXiv)

Effective explanations of video action recognition models should disentangle how movements unfold over time from the surrounding spatial context. However, existing methods based on saliency produce entangled explanations, making it unclear whether predictions rely on motion or spatial context. Language-based approaches offer structure but often fail to explain motions due to their tacit nature -- intuitively understood but difficult to verbalize. To address these challenges, we propose Disentangled Action aNd Context concept-based Explainable (DANCE) video action recognition, a framework that predicts actions through disentangled concept types: motion dynamics, objects, and scenes. We define motion dynamics concepts as human pose sequences. We employ a large language model to automatically extract object and scene concepts. Built on an ante-hoc concept bottleneck design, DANCE enforces prediction through these concepts. Experiments on four datasets -- KTH, Penn Action, HAA500, and UCF-101 -- demonstrate that DANCE significantly improves explanation clarity with competitive performance. We validate the superior interpretability of DANCE through a user study. Experimental results also show that DANCE is beneficial for model debugging, editing, and failure analysis.