# Learning Video Representations from Correspondence Proposals

arXiv: https://arxiv.org/abs/1905.07853 · HTML: https://mlsanity.com/paper/1905.07853/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2019.

## Built on by (most cited first, 22 of 22)

- [ACTION-Net: Multipath Excitation for Action Recognition](https://mlsanity.com/paper/2103.07372/) (2021)
- [StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition](https://mlsanity.com/paper/2503.06220/) (2025)
- [Learning Self-Similarity in Space and Time as Generalized Motion for Video Action Recognition](https://mlsanity.com/paper/2102.07092/) (2021)
- [Stand-Alone Inter-Frame Attention in Video Models](https://mlsanity.com/paper/2206.06931/) (2022)
- [Unsupervised Learning of Accurate Siamese Tracking](https://mlsanity.com/paper/2204.01475/) (2022)
- [Relational Self-Attention: What's Missing in Attention for Video Understanding](https://mlsanity.com/paper/2111.01673/) (2021)
- [Learning Correlation Structures for Vision Transformers](https://mlsanity.com/paper/2404.03924/) (2024)
- [United We Stand, Divided We Fall: UnityGraph for Unsupervised Procedure Learning from Videos](https://mlsanity.com/paper/2311.03550/) (2023)
- [KDFNet: Learning Keypoint Distance Field for 6D Object Pose Estimation](https://mlsanity.com/paper/2109.10127/) (2021)
- [Motion-driven Visual Tempo Learning for Video-based Action Recognition](https://mlsanity.com/paper/2202.12116/) (2022)
- [Dynamic Temporal Filtering in Video Models](https://mlsanity.com/paper/2211.08252/) (2022)
- [Spatiotemporal Deformable Scene Graphs for Complex Activity Detection](https://mlsanity.com/paper/2104.08194/) (2021)
- [Weakly-Supervised Temporal Action Localization by Progressive Complementary Learning](https://mlsanity.com/paper/2206.11011/) (2022)
- [Cross-Modal Graph with Meta Concepts for Video Captioning](https://mlsanity.com/paper/2108.06458/) (2021)
- [Bi-Calibration Networks for Weakly-Supervised Video Representation Learning](https://mlsanity.com/paper/2206.10491/) (2022)
- [What's in the Flow? Exploiting Temporal Motion Cues for Unsupervised Generic Event Boundary Detection](https://mlsanity.com/paper/2404.18935/) (2024)
- [Motion Aware Self-Supervision for Generic Event Boundary Detection](https://mlsanity.com/paper/2210.05574/) (2022)
- [Rich Action-semantic Consistent Knowledge for Early Action Prediction](https://mlsanity.com/paper/2201.09169/) (2022)
- [Position-aware Location Regression Network for Temporal Video Grounding](https://mlsanity.com/paper/2204.05499/) (2022)
- [FuTH-Net: Fusing Temporal Relations and Holistic Features for Aerial Video Classification](https://mlsanity.com/paper/2209.11316/) (2022)
- [Leveraging Text-to-Image Diffusion Models for Unsupervised Visual Object Tracking](https://mlsanity.com/paper/2605.26933/) (2026)
- [ST-ABN: Visual Explanation Taking into Account Spatio-temporal Information for Video Recognition](https://mlsanity.com/paper/2110.15574/) (2021)

## Abstract (arXiv)

Correspondences between frames encode rich information about dynamic content in videos. However, it is challenging to effectively capture and learn those due to their irregular structure and complex dynamics. In this paper, we propose a novel neural network that learns video representations by aggregating information from potential correspondences. This network, named CPNet, can learn evolving 2D fields with temporal consistency. In particular, it can effectively learn representations for videos by mixing appearance and long-range motion with an RGB-only input. We provide extensive ablation experiments to validate our model. CPNet shows stronger performance than existing methods on Kinetics and achieves the state-of-the-art performance on Something-Something and Jester. We provide analysis towards the behavior of our model and show its robustness to errors in proposals.