# Fusing Multi-Stream Deep Networks for Video Classification

arXiv: https://arxiv.org/abs/1509.06086 · HTML: https://mlsanity.com/paper/1509.06086/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2015.

## Built on by (most cited first, 4 of 4)

- [Event and Activity Recognition in Video Surveillance for Cyber-Physical Systems](https://mlsanity.com/paper/2111.02064/) (2021)
- [Learning Muti-expert Distribution Calibration for Long-tailed Video Classification](https://mlsanity.com/paper/2205.10788/) (2022)
- [A Multimodal Sentiment Dataset for Video Recommendation](https://mlsanity.com/paper/2109.08333/) (2021)
- [MsEdF: A Multi-stream Encoder-decoder Framework for Remote Sensing Image Captioning](https://mlsanity.com/paper/2502.09282/) (2025)

## Abstract (arXiv)

This paper studies deep network architectures to address the problem of video classification. A multi-stream framework is proposed to fully utilize the rich multimodal information in videos. Specifically, we first train three Convolutional Neural Networks to model spatial, short-term motion and audio clues respectively. Long Short Term Memory networks are then adopted to explore long-term temporal dynamics. With the outputs of the individual streams, we propose a simple and effective fusion method to generate the final predictions, where the optimal fusion weights are learned adaptively for each class, and the learning process is regularized by automatically estimated class relationships. Our contributions are two-fold. First, the proposed multi-stream framework is able to exploit multimodal features that are more comprehensive than those previously attempted. Second, we demonstrate that the adaptive fusion method using the class relationship as a regularizer outperforms traditional alternatives that estimate the weights in a "free" fashion. Our framework produces significantly better results than the state of the arts on two popular benchmarks, 92.2% on UCF-101 (without using audio) and 84.9% on Columbia Consumer Videos.