# Locality-Aware Inter-and Intra-Video Reconstruction for Self-Supervised Correspondence Learning

arXiv: https://arxiv.org/abs/2203.14333 · HTML: https://mlsanity.com/paper/2203.14333/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 79 of its 109 references that are in the map.
- First posted on arXiv 27 Mar 2022.

## Builds on (most cited first, 30 of 79)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [End-to-End Learning of Visual Representations from Uncurated Instructional Videos](https://mlsanity.com/paper/1912.06430/) (2019)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [ActBERT: Learning Global-Local Video-Text Representations](https://mlsanity.com/paper/2011.07231/) (2020)
- [Learning Representations for Automatic Colorization](https://mlsanity.com/paper/1603.06668/) (2016)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [Self-supervised Co-training for Video Representation Learning](https://mlsanity.com/paper/2010.09709/) (2020)

## Built on by (most cited first, 26 of 26)

- [Global-to-Local Modeling for Video-based 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2303.14747/) (2023)
- [Advances in Deep Concealed Scene Understanding](https://mlsanity.com/paper/2304.11234/) (2023)
- [Boosting Video Object Segmentation via Space-time Correspondence Learning](https://mlsanity.com/paper/2304.06211/) (2023)
- [VideoMAC: Video Masked Autoencoders Meet ConvNets](https://mlsanity.com/paper/2402.19082/) (2024)
- [Deep Learning Technique for Human Parsing: A Survey and Outlook](https://mlsanity.com/paper/2301.00394/) (2023)
- [Unified Mask Embedding and Correspondence Learning for Self-Supervised Video Segmentation](https://mlsanity.com/paper/2303.10100/) (2023)
- [Spatial-then-Temporal Self-Supervised Learning for Video Correspondence](https://mlsanity.com/paper/2209.07778/) (2022)
- [Clustering Propagation for Universal Medical Image Segmentation](https://mlsanity.com/paper/2403.16646/) (2024)
- [Predicting Video Slot Attention Queries from Random Slot-Feature Pairs](https://mlsanity.com/paper/2508.01345/) (2025)
- [Multimodal Large Models Are Effective Action Anticipators](https://mlsanity.com/paper/2501.00795/) (2025)
- [General and Task-Oriented Video Segmentation](https://mlsanity.com/paper/2407.06540/) (2024)
- [Learning Fine-Grained Features for Pixel-wise Video Correspondences](https://mlsanity.com/paper/2308.03040/) (2023)
- [UniParser: Multi-Human Parsing with Unified Correlation Representation Learning](https://mlsanity.com/paper/2310.08984/) (2023)
- [DiDA: Video Object Segmentation with Distillation Learning of Deformable Attention](https://mlsanity.com/paper/2401.13937/) (2024)
- [SGM-Net: Semantic Guided Matting Net](https://mlsanity.com/paper/2208.07496/) (2022)
- [TIVE: A Toolbox for Identifying Video Instance Segmentation Errors](https://mlsanity.com/paper/2210.08856/) (2022)
- [Deep Learning for Human Parsing: A Survey](https://mlsanity.com/paper/2301.12416/) (2023)
- [ZJU ReLER Submission for EPIC-KITCHEN Challenge 2023: TREK-150 Single Object Tracking](https://mlsanity.com/paper/2307.02508/) (2023)
- [ZJU ReLER Submission for EPIC-KITCHEN Challenge 2023: Semi-Supervised Video Object Segmentation](https://mlsanity.com/paper/2307.02010/) (2023)
- [Latent Spatiotemporal Adaptation for Generalized Face Forgery Video Detection](https://mlsanity.com/paper/2309.04795/) (2023)
- [OneSeg: Self-learning and One-shot Learning based Single-slice Annotation for 3D Medical Image Segmentation](https://mlsanity.com/paper/2309.13671/) (2023)
- [Dynamic in Static: Hybrid Visual Correspondence for Self-Supervised Video Object Segmentation](https://mlsanity.com/paper/2404.13505/) (2024)
- [Leveraging Motion Information for Better Self-Supervised Video Correspondence Learning](https://mlsanity.com/paper/2503.12026/) (2025)
- [Enhancing Self-Supervised Fine-Grained Video Object Tracking with Dynamic Memory Prediction](https://mlsanity.com/paper/2504.21692/) (2025)
- [Learning 3D Texture-Aware Representations for Parsing Diverse Human Clothing and Body Parts](https://mlsanity.com/paper/2508.06032/) (2025)
- [SWT: Self-Supervised Video Object Segmentation via Sliding, Wavelet and Transportation](https://mlsanity.com/paper/2609.31725/) (2026)

## Abstract (arXiv)

Our target is to learn visual correspondence from unlabeled videos. We develop LIIR, a locality-aware inter-and intra-video reconstruction framework that fills in three missing pieces, i.e., instance discrimination, location awareness, and spatial compactness, of self-supervised correspondence learning puzzle. First, instead of most existing efforts focusing on intra-video self-supervision only, we exploit cross video affinities as extra negative samples within a unified, inter-and intra-video reconstruction scheme. This enables instance discriminative representation learning by contrasting desired intra-video pixel association against negative inter-video correspondence. Second, we merge position information into correspondence matching, and design a position shifting strategy to remove the side-effect of position encoding during inter-video affinity computation, making our LIIR location-sensitive. Third, to make full use of the spatial continuity nature of video data, we impose a compactness-based constraint on correspondence matching, yielding more sparse and reliable solutions. The learned representation surpasses self-supervised state-of-the-arts on label propagation tasks including objects, semantic parts, and keypoints.