# Interpreting video features: a comparison of 3D convolutional networks and convolutional LSTM networks

arXiv: https://arxiv.org/abs/2002.00367 · HTML: https://mlsanity.com/paper/2002.00367/

## Key facts

- Cited by 12 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 2 Feb 2020.

## Built on by (most cited first, 12 of 12)

- [Mitigating and Evaluating Static Bias of Action Representations in the Background and the Foreground](https://mlsanity.com/paper/2211.12883/) (2022)
- [A Deeper Dive Into What Deep Spatiotemporal Networks Encode: Quantifying Static vs. Dynamic Information](https://mlsanity.com/paper/2206.02846/) (2022)
- [Is Appearance Free Action Recognition Possible?](https://mlsanity.com/paper/2207.06261/) (2022)
- [Recur, Attend or Convolve? On Whether Temporal Modeling Matters for Cross-Domain Robustness in Action Recognition](https://mlsanity.com/paper/2112.12175/) (2021)
- [Quantifying and Learning Static vs. Dynamic Information in Deep Spatiotemporal Networks](https://mlsanity.com/paper/2211.01783/) (2022)
- [An Integrated Framework for Multi-Granular Explanation of Video Summarization](https://mlsanity.com/paper/2405.10082/) (2024)
- [Process-of-Thought Reasoning for Videos](https://mlsanity.com/paper/2602.07689/) (2026)
- [Encoder-Decoder Networks for Analyzing Thermal and Power Delivery Networks](https://mlsanity.com/paper/2110.14197/) (2021)
- [Supervised contrastive learning for cell stage classification of animal embryos](https://mlsanity.com/paper/2502.07360/) (2025)
- [Ground Reaction Force Estimation via Time-aware Knowledge Distillation](https://mlsanity.com/paper/2506.10265/) (2025)
- [An Experimental Study on Generating Plausible Textual Explanations for Video Summarization](https://mlsanity.com/paper/2509.26225/) (2025)
- [Selective Correlation Based Knowledge Distillation for Ground Reaction Force Estimation](https://mlsanity.com/paper/2605.00888/) (2026)

## Abstract (arXiv)

A number of techniques for interpretability have been presented for deep learning in computer vision, typically with the goal of understanding what the networks have based their classification on. However, interpretability for deep video architectures is still in its infancy and we do not yet have a clear concept of how to decode spatiotemporal features. In this paper, we present a study comparing how 3D convolutional networks and convolutional LSTM networks learn features across temporally dependent frames. This is the first comparison of two video models that both convolve to learn spatial features but have principally different methods of modeling time. Additionally, we extend the concept of meaningful perturbation introduced by citeMeaningFulPert to the temporal dimension, to identify the temporal part of a sequence most meaningful to the network for a classification decision. Our findings indicate that the 3D convolutional model concentrates on shorter events in the input sequence, and places its spatial focus on fewer, contiguous areas.