# SWTF: Sparse Weighted Temporal Fusion for Drone-Based Activity Recognition

arXiv: https://arxiv.org/abs/2211.05531 · HTML: https://mlsanity.com/paper/2211.05531/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 10 Nov 2022.

## Builds on (most cited first, 18 of 18)

- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [ECO: Efficient Convolutional Network for Online Video Understanding](https://mlsanity.com/paper/1804.09066/) (2018)
- [ActionVLAD: Learning spatio-temporal aggregation for action classification](https://mlsanity.com/paper/1704.02895/) (2017)
- [VideoLSTM Convolves, Attends and Flows for Action Recognition](https://mlsanity.com/paper/1607.01794/) (2016)
- [P-CNN: Pose-based CNN Features for Action Recognition](https://mlsanity.com/paper/1506.03607/) (2015)
- [Okutama-Action: An Aerial View Video Dataset for Concurrent Human Action Detection](https://mlsanity.com/paper/1706.03038/) (2017)
- [Towards Good Practices for Very Deep Two-Stream ConvNets](https://mlsanity.com/paper/1507.02159/) (2015)
- [Im2Flow: Motion Hallucination from Static Images for Action Recognition](https://mlsanity.com/paper/1712.04109/) (2017)
- [PAN: Towards Fast Action Recognition via Learning Persistence of Appearance](https://mlsanity.com/paper/2008.03462/) (2020)
- [Non-Linear Temporal Subspace Representations for Activity Recognition](https://mlsanity.com/paper/1803.11064/) (2018)
- [A Multi-viewpoint Outdoor Dataset for Human Action Recognition](https://mlsanity.com/paper/2110.04119/) (2021)
- [Temporal-Spatial Mapping for Action Recognition](https://mlsanity.com/paper/1809.03669/) (2018)

## Abstract (arXiv)

Drone-camera based human activity recognition (HAR) has received significant attention from the computer vision research community in the past few years. A robust and efficient HAR system has a pivotal role in fields like video surveillance, crowd behavior analysis, sports analysis, and human-computer interaction. What makes it challenging are the complex poses, understanding different viewpoints, and the environmental scenarios where the action is taking place. To address such complexities, in this paper, we propose a novel Sparse Weighted Temporal Fusion (SWTF) module to utilize sparsely sampled video frames for obtaining global weighted temporal fusion outcome. The proposed SWTF is divided into two components. First, a temporal segment network that sparsely samples a given set of frames. Second, weighted temporal fusion, that incorporates a fusion of feature maps derived from optical flow, with raw RGB images. This is followed by base-network, which comprises a convolutional neural network module along with fully connected layers that provide us with activity recognition. The SWTF network can be used as a plug-in module to the existing deep CNN architectures, for optimizing them to learn temporal information by eliminating the need for a separate temporal stream. It has been evaluated on three publicly available benchmark datasets, namely Okutama, MOD20, and Drone-Action. The proposed model has received an accuracy of 72.76%, 92.56%, and 78.86% on the respective datasets thereby surpassing the previous state-of-the-art performances by a significant margin.