# Continual Learning of Conjugated Visual Representations through Higher-order Motion Flows

arXiv: https://arxiv.org/abs/2409.11441 · HTML: https://mlsanity.com/paper/2409.11441/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 57 references that are in the map.
- First posted on arXiv 16 Sep 2024.

## Builds on (most cited first, 30 of 34)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [Recent Advances in Open Set Recognition: A Survey](https://mlsanity.com/paper/1811.08581/) (2018)
- [Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2011.10043/) (2020)
- [Task-Free Continual Learning](https://mlsanity.com/paper/1812.03596/) (2018)
- [Continual Unsupervised Representation Learning](https://mlsanity.com/paper/1910.14481/) (2019)
- [Learning Features by Watching Objects Move](https://mlsanity.com/paper/1612.06370/) (2016)
- [Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System](https://mlsanity.com/paper/2201.12604/) (2022)
- [Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinement](https://mlsanity.com/paper/2010.07958/) (2020)
- [Representational Continuity for Unsupervised Continual Learning](https://mlsanity.com/paper/2110.06976/) (2021)
- [Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning](https://mlsanity.com/paper/2101.08482/) (2021)
- [How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2203.16262/) (2022)
- [Self-Supervised Representation Learning from Flow Equivariance](https://mlsanity.com/paper/2101.06553/) (2021)
- [Cross Pixel Optical Flow Similarity for Self-Supervised Learning](https://mlsanity.com/paper/1807.05636/) (2018)
- [Delta Distillation for Efficient Video Processing](https://mlsanity.com/paper/2203.09594/) (2022)
- [SAILenv: Learning in Virtual Visual Environments Made Simple](https://mlsanity.com/paper/2007.08224/) (2020)

## Abstract (arXiv)

Learning with neural networks from a continuous stream of visual information presents several challenges due to the non-i.i.d. nature of the data. However, it also offers novel opportunities to develop representations that are consistent with the information flow. In this paper we investigate the case of unsupervised continual learning of pixel-wise features subject to multiple motion-induced constraints, therefore named motion-conjugated feature representations. Differently from existing approaches, motion is not a given signal (either ground-truth or estimated by external modules), but is the outcome of a progressive and autonomous learning process, occurring at various levels of the feature hierarchy. Multiple motion flows are estimated with neural networks and characterized by different levels of abstractions, spanning from traditional optical flow to other latent signals originating from higher-level features, hence called higher-order motions. Continuously learning to develop consistent multi-order flows and representations is prone to trivial solutions, which we counteract by introducing a self-supervised contrastive loss, spatially-aware and based on flow-induced similarity. We assess our model on photorealistic synthetic streams and real-world videos, comparing to pre-trained state-of-the art feature extractors (also based on Transformers) and to recent unsupervised learning models, significantly outperforming these alternatives.