# Appearance-and-Relation Networks for Video Classification

arXiv: https://arxiv.org/abs/1711.09125 · HTML: https://mlsanity.com/paper/1711.09125/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2017.

## Built on by (most cited first, 30 of 61)

- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [Relaxed Transformer Decoders for Direct Action Proposal Generation](https://mlsanity.com/paper/2102.01894/) (2021)
- [Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation](https://mlsanity.com/paper/2303.00440/) (2023)
- [Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding](https://mlsanity.com/paper/2109.04872/) (2021)
- [MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actions](https://mlsanity.com/paper/2105.07404/) (2021)
- [TALLFormer: Temporal Action Localization with a Long-memory Transformer](https://mlsanity.com/paper/2204.01680/) (2022)
- [Revisiting Classifier: Transferring Vision-Language Models for Video Recognition](https://mlsanity.com/paper/2207.01297/) (2022)
- [Target Adaptive Context Aggregation for Video Scene Graph Generation](https://mlsanity.com/paper/2108.08121/) (2021)
- [Contrastive Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2211.09980/) (2022)
- [Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models](https://mlsanity.com/paper/2301.00182/) (2022)
- [MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition](https://mlsanity.com/paper/2108.09322/) (2021)
- [Weakly Supervised Video Anomaly Detection and Localization with Spatio-Temporal Prompts](https://mlsanity.com/paper/2408.05905/) (2024)
- [Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning](https://mlsanity.com/paper/2309.07911/) (2023)
- [Aligning Correlation Information for Domain Adaptation in Action Recognition](https://mlsanity.com/paper/2107.04932/) (2021)
- [MAR: Masked Autoencoders for Efficient Action Recognition](https://mlsanity.com/paper/2207.11660/) (2022)
- [BasicTAD: an Astounding RGB-Only Baseline for Temporal Action Detection](https://mlsanity.com/paper/2205.02717/) (2022)
- [MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition](https://mlsanity.com/paper/2305.07214/) (2023)
- [VAU-R1: Advancing Video Anomaly Understanding via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2505.23504/) (2025)
- [When Video Classification Meets Incremental Classes](https://mlsanity.com/paper/2106.15827/) (2021)
- [Masked Video and Body-worn IMU Autoencoder for Egocentric Action Recognition](https://mlsanity.com/paper/2407.06628/) (2024)
- [RGB Stream Is Enough for Temporal Action Detection](https://mlsanity.com/paper/2107.04362/) (2021)
- [EAN: Event Adaptive Network for Enhanced Action Recognition](https://mlsanity.com/paper/2107.10771/) (2021)
- [End-to-End Compressed Video Representation Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2203.15336/) (2022)
- [Cross-Enhancement Transformer for Action Segmentation](https://mlsanity.com/paper/2205.09445/) (2022)
- [Temporal Perceiver: A General Architecture for Arbitrary Boundary Detection](https://mlsanity.com/paper/2203.00307/) (2022)
- [TFCNet: Temporal Fully Connected Networks for Static Unbiased Temporal Reasoning](https://mlsanity.com/paper/2203.05928/) (2022)
- [Temporal-attentive Covariance Pooling Networks for Video Recognition](https://mlsanity.com/paper/2110.14381/) (2021)
- [Learning a Condensed Frame for Memory-Efficient Video Class-Incremental Learning](https://mlsanity.com/paper/2211.00833/) (2022)
- [Motion-driven Visual Tempo Learning for Video-based Action Recognition](https://mlsanity.com/paper/2202.12116/) (2022)
- [Uni-AdaFocus: Spatial-temporal Dynamic Computation for Video Recognition](https://mlsanity.com/paper/2412.11228/) (2024)

## Abstract (arXiv)

Spatiotemporal feature learning in videos is a fundamental problem in computer vision. This paper presents a new architecture, termed as Appearance-and-Relation Network (ARTNet), to learn video representation in an end-to-end manner. ARTNets are constructed by stacking multiple generic building blocks, called as SMART, whose goal is to simultaneously model appearance and relation from RGB input in a separate and explicit manner. Specifically, SMART blocks decouple the spatiotemporal learning module into an appearance branch for spatial modeling and a relation branch for temporal modeling. The appearance branch is implemented based on the linear combination of pixels or filter responses in each frame, while the relation branch is designed based on the multiplicative interactions between pixels or filter responses across multiple frames. We perform experiments on three action recognition benchmarks: Kinetics, UCF101, and HMDB51, demonstrating that SMART blocks obtain an evident improvement over 3D convolutions for spatiotemporal feature learning. Under the same training setting, ARTNets achieve superior performance on these three datasets to the existing state-of-the-art methods.