# LoCo: Local Contrastive Representation Learning

arXiv: https://arxiv.org/abs/2008.01342 · HTML: https://mlsanity.com/paper/2008.01342/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Aug 2020.

## Built on by (most cited first, 30 of 55)

- [SpectralGPT: Spectral Remote Sensing Foundation Model](https://mlsanity.com/paper/2311.07113/) (2023)
- [CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow](https://mlsanity.com/paper/2211.10408/) (2022)
- [Efficient Self-supervised Vision Transformers for Representation Learning](https://mlsanity.com/paper/2106.09785/) (2021)
- [Scaling Forward Gradient With Local Losses](https://mlsanity.com/paper/2210.03310/) (2022)
- [Understanding Masked Image Modeling via Learning Occlusion Invariant Feature](https://mlsanity.com/paper/2208.04164/) (2022)
- [Training of Physical Neural Networks](https://mlsanity.com/paper/2406.03372/) (2024)
- [You Never Cluster Alone](https://mlsanity.com/paper/2106.01908/) (2021)
- [DreamTeacher: Pretraining Image Backbones with Deep Generative Models](https://mlsanity.com/paper/2307.07487/) (2023)
- [Self-Supervised Representation Learning from Flow Equivariance](https://mlsanity.com/paper/2101.06553/) (2021)
- [Enhancing Self-supervised Video Representation Learning via Multi-level Feature Optimization](https://mlsanity.com/paper/2108.02183/) (2021)
- [Blockwise Self-Supervised Learning at Scale](https://mlsanity.com/paper/2302.01647/) (2023)
- [Trustworthy Recommender Systems](https://mlsanity.com/paper/2208.06265/) (2022)
- [Contrastive Learning for Prompt-Based Few-Shot Language Learners](https://mlsanity.com/paper/2205.01308/) (2022)
- [STC: Spatio-Temporal Contrastive Learning for Video Instance Segmentation](https://mlsanity.com/paper/2202.03747/) (2022)
- [Meta-Learning Bidirectional Update Rules](https://mlsanity.com/paper/2104.04657/) (2021)
- [Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2209.09433/) (2022)
- [Momentum Auxiliary Network for Supervised Local Learning](https://mlsanity.com/paper/2407.05623/) (2024)
- [Rethinking Prototypical Contrastive Learning through Alignment, Uniformity and Correlation](https://mlsanity.com/paper/2210.10194/) (2022)
- [Scaling Supervised Local Learning with Augmented Auxiliary Networks](https://mlsanity.com/paper/2402.17318/) (2024)
- [End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training](https://mlsanity.com/paper/2402.09050/) (2024)
- [Distance-Forward Learning: Enhancing the Forward-Forward Algorithm Towards High-Performance On-Chip Learning](https://mlsanity.com/paper/2408.14925/) (2024)
- [Aggregating Capacity in FL through Successive Layer Training for Computationally-Constrained Devices](https://mlsanity.com/paper/2305.17005/) (2023)
- [HEX: Hierarchical Emergence Exploitation in Self-Supervised Algorithms](https://mlsanity.com/paper/2410.23200/) (2024)
- [Contrastive Learning of Global-Local Video Representations](https://mlsanity.com/paper/2104.05418/) (2021)
- [Block-local learning with probabilistic latent representations](https://mlsanity.com/paper/2305.14974/) (2023)
- [Advancing Supervised Local Learning Beyond Classification with Long-term Feature Bank](https://mlsanity.com/paper/2406.00446/) (2024)
- [Self-Contrastive Learning: Single-viewed Supervised Contrastive Framework using Sub-network](https://mlsanity.com/paper/2106.15499/) (2021)
- [MLAAN: Scaling Supervised Local Learning with Multilaminar Leap Augmented Auxiliary Network](https://mlsanity.com/paper/2406.16633/) (2024)
- [HPFF: Hierarchical Locally Supervised Learning with Patch Feature Fusion](https://mlsanity.com/paper/2407.05638/) (2024)
- [Unsupervised Disentanglement without Autoencoding: Pitfalls and Future Directions](https://mlsanity.com/paper/2108.06613/) (2021)

## Abstract (arXiv)

Deep neural nets typically perform end-to-end backpropagation to learn the weights, a procedure that creates synchronization constraints in the weight update step across layers and is not biologically plausible. Recent advances in unsupervised contrastive representation learning point to the question of whether a learning algorithm can also be made local, that is, the updates of lower layers do not directly depend on the computation of upper layers. While Greedy InfoMax separately learns each block with a local objective, we found that it consistently hurts readout accuracy in state-of-the-art unsupervised contrastive learning algorithms, possibly due to the greedy objective as well as gradient isolation. In this work, we discover that by overlapping local blocks stacking on top of each other, we effectively increase the decoder depth and allow upper blocks to implicitly send feedbacks to lower blocks. This simple design closes the performance gap between local learning and end-to-end contrastive learning algorithms for the first time. Aside from standard ImageNet experiments, we also show results on complex downstream tasks such as object detection and instance segmentation directly using readout features.