# A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks

arXiv: https://arxiv.org/abs/2007.00800 · HTML: https://mlsanity.com/paper/2007.00800/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2020.

## Built on by (most cited first, 18 of 18)

- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Self-supervised learning methods and applications in medical imaging analysis: A survey](https://mlsanity.com/paper/2109.08685/) (2021)
- [A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trends](https://mlsanity.com/paper/2410.15067/) (2024)
- [Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data](https://mlsanity.com/paper/2206.02353/) (2022)
- [Techniques for Symbol Grounding with SATNet](https://mlsanity.com/paper/2106.11072/) (2021)
- [MedJEx: A Medical Jargon Extraction Model with Wiki's Hyperlink Span and Contextualized Masked Language Model Score](https://mlsanity.com/paper/2210.05875/) (2022)
- [A Survey of the Self Supervised Learning Mechanisms for Vision Transformers](https://mlsanity.com/paper/2408.17059/) (2024)
- [A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques](https://mlsanity.com/paper/2302.08571/) (2023)
- [Self-supervised learning-based general laboratory progress pretrained model for cardiovascular event detection](https://mlsanity.com/paper/2303.06980/) (2023)
- [Re-entry Prediction for Online Conversations via Self-Supervised Learning](https://mlsanity.com/paper/2109.02020/) (2021)
- [Vision-Guided Forecasting -- Visual Context for Multi-Horizon Time Series Forecasting](https://mlsanity.com/paper/2107.12674/) (2021)
- [Soft-labeling Strategies for Rapid Sub-Typing](https://mlsanity.com/paper/2209.12684/) (2022)
- [Memorization of Named Entities in Fine-tuned BERT Models](https://mlsanity.com/paper/2212.03749/) (2022)
- [Cross-Language Speech Emotion Recognition Using Multimodal Dual Attention Transformers](https://mlsanity.com/paper/2306.13804/) (2023)
- [Semi-self-supervised Automated ICD Coding](https://mlsanity.com/paper/2205.10088/) (2022)
- [Soft Labels for Rapid Satellite Object Detection](https://mlsanity.com/paper/2212.00585/) (2022)
- [Examining the Effect of Pre-training on Time Series Classification](https://mlsanity.com/paper/2309.05256/) (2023)
- [Is Self-Pretraining really useful to improve diagnosis in medical Time Series?](https://mlsanity.com/paper/2608.06122/) (2026)

## Abstract (arXiv)

Deep neural networks are typically trained under a supervised learning framework where a model learns a single task using labeled data. Instead of relying solely on labeled data, practitioners can harness unlabeled or related data to improve model performance, which is often more accessible and ubiquitous. Self-supervised pre-training for transfer learning is becoming an increasingly popular technique to improve state-of-the-art results using unlabeled data. It involves first pre-training a model on a large amount of unlabeled data, then adapting the model to target tasks of interest. In this review, we survey self-supervised learning methods and their applications within the sequential transfer learning framework. We provide an overview of the taxonomy for self-supervised learning and transfer learning, and highlight some prominent methods for designing pre-training tasks across different domains. Finally, we discuss recent trends and suggest areas for future investigation.