# Concepts in Motion: Temporal Concept Bottleneck Model for Interpretable Video Classification

arXiv: https://arxiv.org/abs/2509.20899 · HTML: https://mlsanity.com/paper/2509.20899/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 52 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [Perception Encoder: The best visual embeddings are not at the output of the network](https://mlsanity.com/paper/2504.13181/) (2025)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [Label-Free Concept Bottleneck Models](https://mlsanity.com/paper/2304.06129/) (2023)
- [Recurrent Independent Mechanisms](https://mlsanity.com/paper/1909.10893/) (2019)
- [Self-Attention Attribution: Interpreting Information Interactions Inside Transformer](https://mlsanity.com/paper/2004.11207/) (2020)
- [Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectors](https://mlsanity.com/paper/2006.15417/) (2020)
- [Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges](https://mlsanity.com/paper/2010.09337/) (2020)
- [Revisiting Multiple Instance Neural Networks](https://mlsanity.com/paper/1610.02501/) (2016)
- [Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks](https://mlsanity.com/paper/1912.09930/) (2019)
- [Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery](https://mlsanity.com/paper/2407.14499/) (2024)
- [Concept Bottleneck Model with Additional Unsupervised Concepts](https://mlsanity.com/paper/2202.01459/) (2022)
- [Interactive Concept Bottleneck Models](https://mlsanity.com/paper/2212.07430/) (2022)
- [Attention over learned object embeddings enables complex visual reasoning](https://mlsanity.com/paper/2012.08508/) (2020)
- [Concept Bottleneck Large Language Models](https://mlsanity.com/paper/2412.07992/) (2024)
- [HAA500: Human-Centric Atomic Action Dataset with Curated Videos](https://mlsanity.com/paper/2009.05224/) (2020)
- [Planning with Reasoning using Vision Language World Model](https://mlsanity.com/paper/2509.02722/) (2025)
- [No frame left behind: Full Video Action Recognition](https://mlsanity.com/paper/2103.15395/) (2021)
- [Understanding Video Transformers via Universal Concept Discovery](https://mlsanity.com/paper/2401.10831/) (2024)

## Built on by (most cited first, 1 of 1)

- [TRACE: A Concept Bottleneck Model for Longitudinal 3D Glioblastoma Response Assessment](https://mlsanity.com/paper/2606.30313/) (2026)

## Abstract (arXiv)

Concept Bottleneck Models (CBMs) enable interpretable image classification by structuring predictions around human-understandable concepts, but extending this paradigm to video remains challenging due to the difficulty of extracting concepts and modeling them over time. In this paper, we introduce MoTIF (Moving Temporal Interpretable Framework), a transformer-based concept architecture that operates on sequences of temporally grounded concept activations, by employing per-concept temporal self-attention to model when individual concepts recur and how their temporal patterns contribute to predictions. Central to the framework is a class-conditioned VLM-based concept discovery module that extracts object- and action-centric textual concepts from training videos, yielding temporally expressive concept sets without manual concept annotation. Across multiple video benchmarks, this combination improves over global concept bottlenecks and remains competitive within the interpretable concept-bottleneck setting, while narrowing the gap to strong black-box video baselines that we report as contextual references. Code available at github.com/patrick-knab/MoTIF.