# Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks

arXiv: https://arxiv.org/abs/1502.07209 · HTML: https://mlsanity.com/paper/1502.07209/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Feb 2015.

## Built on by (most cited first, 30 of 45)

- [Adaptive Focus for Efficient Video Recognition](https://mlsanity.com/paper/2105.03245/) (2021)
- [Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures](https://mlsanity.com/paper/2307.15220/) (2023)
- [FineAction: A Fine-Grained Video Dataset for Temporal Action Localization](https://mlsanity.com/paper/2105.11107/) (2021)
- [AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognition](https://mlsanity.com/paper/2112.14238/) (2021)
- [AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition](https://mlsanity.com/paper/2105.05165/) (2021)
- [Dynamic Network Quantization for Efficient Video Inference](https://mlsanity.com/paper/2108.10394/) (2021)
- [VideoLT: Large-scale Long-tailed Video Recognition](https://mlsanity.com/paper/2105.02668/) (2021)
- [OCSampler: Compressing Videos to One Clip with Single-step Sampling](https://mlsanity.com/paper/2201.04388/) (2022)
- [Boosting the Transferability of Video Adversarial Examples via Temporal Translation](https://mlsanity.com/paper/2110.09075/) (2021)
- [AdaFocusV3: On Unified Spatial-temporal Dynamic Video Recognition](https://mlsanity.com/paper/2209.13465/) (2022)
- [SpotEM: Efficient Video Search for Episodic Memory](https://mlsanity.com/paper/2306.15850/) (2023)
- [NSNet: Non-saliency Suppression Sampler for Efficient Video Recognition](https://mlsanity.com/paper/2207.10388/) (2022)
- [Contrastive Masked Autoencoders for Self-Supervised Video Hashing](https://mlsanity.com/paper/2211.11210/) (2022)
- [Anomaly Detection in Video Sequences: A Benchmark and Computational Model](https://mlsanity.com/paper/2106.08570/) (2021)
- [Dynamic Enhancement Network for Partial Multi-modality Person Re-identification](https://mlsanity.com/paper/2305.15762/) (2023)
- [Temporal Saliency Query Network for Efficient Video Recognition](https://mlsanity.com/paper/2207.10379/) (2022)
- [Uni-AdaFocus: Spatial-temporal Dynamic Computation for Video Recognition](https://mlsanity.com/paper/2412.11228/) (2024)
- [Cross-Modal Transferable Image-to-Video Attack on Video Quality Metrics](https://mlsanity.com/paper/2501.08415/) (2025)
- [Weakly-Supervised Temporal Action Detection for Fine-Grained Videos with Hierarchical Atomic Actions](https://mlsanity.com/paper/2207.11805/) (2022)
- [Rethinking Resolution in the Context of Efficient Video Recognition](https://mlsanity.com/paper/2209.12797/) (2022)
- [Survey: Exploiting Data Redundancy for Optimization of Deep Learning](https://mlsanity.com/paper/2208.13363/) (2022)
- [Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training](https://mlsanity.com/paper/2303.13635/) (2023)
- [Label-only Model Inversion Attack: The Attack that Requires the Least Information](https://mlsanity.com/paper/2203.06555/) (2022)
- [Fine-grained Activities of People Worldwide](https://mlsanity.com/paper/2207.05182/) (2022)
- [ViGAT: Bottom-up event recognition and explanation in video using factorized graph attention network](https://mlsanity.com/paper/2207.09927/) (2022)
- [View while Moving: Efficient Video Recognition in Long-untrimmed Videos](https://mlsanity.com/paper/2308.04834/) (2023)
- [Audio-Visual Glance Network for Efficient Video Recognition](https://mlsanity.com/paper/2308.09322/) (2023)
- [Dual-Stream Knowledge-Preserving Hashing for Unsupervised Video Retrieval](https://mlsanity.com/paper/2310.08009/) (2023)
- [Modeling Temporal Concept Receptive Field Dynamically for Untrimmed Video Analysis](https://mlsanity.com/paper/2111.11653/) (2021)
- [Search-Map-Search: A Frame Selection Paradigm for Action Recognition](https://mlsanity.com/paper/2304.10316/) (2023)

## Abstract (arXiv)

In this paper, we study the challenging problem of categorizing videos according to high-level semantics such as the existence of a particular human action or a complex event. Although extensive efforts have been devoted in recent years, most existing works combined multiple video features using simple fusion strategies and neglected the utilization of inter-class semantic relationships. This paper proposes a novel unified framework that jointly exploits the feature relationships and the class relationships for improved categorization performance. Specifically, these two types of relationships are estimated and utilized by rigorously imposing regularizations in the learning process of a deep neural network (DNN). Such a regularized DNN (rDNN) can be efficiently realized using a GPU-based implementation with an affordable training cost. Through arming the DNN with better capability of harnessing both the feature and the class relationships, the proposed rDNN is more suitable for modeling video semantics. With extensive experimental evaluations, we show that rDNN produces superior performance over several state-of-the-art approaches. On the well-known Hollywood2 and Columbia Consumer Video benchmarks, we obtain very competitive results: 66.9% and 73.5% respectively in terms of mean average precision. In addition, to substantially evaluate our rDNN and stimulate future research on large scale video categorization, we collect and release a new benchmark dataset, called FCVID, which contains 91,223 Internet videos and 239 manually annotated categories.