# Leveraging Random Label Memorization for Unsupervised Pre-Training

arXiv: https://arxiv.org/abs/1811.01640 · HTML: https://mlsanity.com/paper/1811.01640/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2018.

## Built on by (most cited first, 12 of 12)

- [Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://mlsanity.com/paper/2205.10770/) (2022)
- [To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Models](https://mlsanity.com/paper/2405.03097/) (2024)
- [Memorization in deep learning: A survey](https://mlsanity.com/paper/2406.03880/) (2024)
- [The Curious Case of Benign Memorization](https://mlsanity.com/paper/2210.14019/) (2022)
- [Compositional generalization in semantic parsing with pretrained transformers](https://mlsanity.com/paper/2109.15101/) (2021)
- [The Vendiscope: An Algorithmic Microscope For Data Collections](https://mlsanity.com/paper/2502.10828/) (2025)
- [Compiled Models, Built-In Exploits: Uncovering Pervasive Bit-Flip Attack Surfaces in DNN Executables](https://mlsanity.com/paper/2309.06223/) (2023)
- [UnrealNAS: Can We Search Neural Architectures with Unreal Data?](https://mlsanity.com/paper/2205.02162/) (2022)
- [Unsupervised Learning of Initialization in Deep Neural Networks via Maximum Mean Discrepancy](https://mlsanity.com/paper/2302.04369/) (2023)
- [On information captured by neural networks: connections with memorization and generalization](https://mlsanity.com/paper/2306.15918/) (2023)
- [Is Supervised Learning Really That Different from Unsupervised?](https://mlsanity.com/paper/2505.11006/) (2025)
- [Localizing Transfer Between Memorization Tasks](https://mlsanity.com/paper/2610.00771/) (2026)

## Abstract (arXiv)

We present a novel approach to leverage large unlabeled datasets by pre-training state-of-the-art deep neural networks on randomly-labeled datasets. Specifically, we train the neural networks to memorize arbitrary labels for all the samples in a dataset and use these pre-trained networks as a starting point for regular supervised learning. Our assumption is that the "memorization infrastructure" learned by the network during the random-label training proves to be beneficial for the conventional supervised learning as well. We test the effectiveness of our pre-training on several video action recognition datasets (HMDB51, UCF101, Kinetics) by comparing the results of the same network with and without the random label pre-training. Our approach yields an improvement - ranging from 1.5% on UCF-101 to 5% on Kinetics - in classification accuracy, which calls for further research in this direction.