# Unsupervised Transfer Learning for Spatiotemporal Predictive Networks

arXiv: https://arxiv.org/abs/2009.11763 · HTML: https://mlsanity.com/paper/2009.11763/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2020.

## Built on by (most cited first, 10 of 10)

- [Domain Adaptation for Time Series Forecasting via Attention Sharing](https://mlsanity.com/paper/2102.06828/) (2021)
- [MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions](https://mlsanity.com/paper/2103.02243/) (2021)
- [Continual Predictive Learning from Videos](https://mlsanity.com/paper/2204.05624/) (2022)
- [MS-RNN: A Flexible Multi-Scale Framework for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2206.03010/) (2022)
- [ModeRNN: Harnessing Spatiotemporal Mode Collapse in Unsupervised Predictive Learning](https://mlsanity.com/paper/2110.03882/) (2021)
- [MS-LSTM: Exploring Spatiotemporal Multiscale Representations in Video Prediction Domain](https://mlsanity.com/paper/2304.07724/) (2023)
- [Unsupervised Unlearning of Concept Drift with Autoencoders](https://mlsanity.com/paper/2211.12989/) (2022)
- [Unsupervised Knowledge Adaptation for Passenger Demand Forecasting](https://mlsanity.com/paper/2206.04053/) (2022)
- [Model-Based Reinforcement Learning with Multi-Task Offline Pretraining](https://mlsanity.com/paper/2306.03360/) (2023)
- [Joint Contrastive-Adversarial Representation Learning for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2610.15613/) (2026)

## Abstract (arXiv)

This paper explores a new research problem of unsupervised transfer learning across multiple spatiotemporal prediction tasks. Unlike most existing transfer learning methods that focus on fixing the discrepancy between supervised tasks, we study how to transfer knowledge from a zoo of unsupervisedly learned models towards another predictive network. Our motivation is that models from different sources are expected to understand the complex spatiotemporal dynamics from different perspectives, thereby effectively supplementing the new task, even if the task has sufficient training samples. Technically, we propose a differentiable framework named transferable memory. It adaptively distills knowledge from a bank of memory states of multiple pretrained RNNs, and applies it to the target network via a novel recurrent structure called the Transferable Memory Unit (TMU). Compared with finetuning, our approach yields significant improvements on three benchmarks for spatiotemporal prediction, and benefits the target task even from less relevant pretext ones.