# Self-supervision through Random Segments with Autoregressive Coding (RandSAC)

arXiv: https://arxiv.org/abs/2203.12054 · HTML: https://mlsanity.com/paper/2203.12054/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 64 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)

## Built on by (most cited first, 16 of 16)

- [SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models](https://mlsanity.com/paper/2305.11281/) (2023)
- [Autoregressive Pretraining with Mamba in Vision](https://mlsanity.com/paper/2406.07537/) (2024)
- [Rejuvenating image-GPT as Strong Visual Representation Learners](https://mlsanity.com/paper/2312.02147/) (2023)
- [Exploring Stochastic Autoregressive Image Modeling for Visual Representation](https://mlsanity.com/paper/2212.01610/) (2022)
- [Masked Modeling for Self-supervised Representation Learning on Vision and Beyond](https://mlsanity.com/paper/2401.00897/) (2023)
- [Medical Vision Generalist: Unifying Medical Imaging Tasks in Context](https://mlsanity.com/paper/2406.05565/) (2024)
- [Obj2Seq: Formatting Objects as Sequences with Class Prompt for Visual Tasks](https://mlsanity.com/paper/2209.13948/) (2022)
- [RoMA: Scaling up Mamba-based Foundation Models for Remote Sensing](https://mlsanity.com/paper/2503.10392/) (2025)
- [Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining](https://mlsanity.com/paper/2407.00935/) (2024)
- [Spatial Entropy as an Inductive Bias for Vision Transformers](https://mlsanity.com/paper/2206.04636/) (2022)
- [ARVideo: Autoregressive Pretraining for Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2405.15160/) (2024)
- [DSeq-JEPA: Discriminative Sequential Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2511.17354/) (2025)
- [Efficient Masked Autoencoders with Self-Consistency](https://mlsanity.com/paper/2302.14431/) (2023)
- [Semantic-Aware Autoregressive Image Modeling for Visual Representation Learning](https://mlsanity.com/paper/2312.10457/) (2023)
- [MAL: Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance](https://mlsanity.com/paper/2412.10730/) (2024)
- [Separators in Enhancing Autoregressive Pretraining for Vision Mamba](https://mlsanity.com/paper/2603.03806/) (2026)

## Abstract (arXiv)

Inspired by the success of self-supervised autoregressive representation learning in natural language (GPT and its variants), and advances in recent visual architecture design with Vision Transformers (ViTs), in this paper, we explore the effect various design choices have on the success of applying such training strategies for visual feature learning. Specifically, we introduce a novel strategy that we call Random Segments with Autoregressive Coding (RandSAC). In RandSAC, we group patch representations (image tokens) into hierarchically arranged segments; within each segment, tokens are predicted in parallel, similar to BERT, while across segment predictions are sequential, similar to GPT. We illustrate that randomized serialization of the segments significantly improves the performance and results in distribution over spatially-long (across-segments) and -short (within-segment) predictions which are effective for feature learning. We illustrate the pertinence of these design choices and explore alternatives on a number of datasets (e.g., CIFAR10, CIFAR100, ImageNet). While our pre-training strategy works with a vanilla Transformer, we also propose a conceptually simple, but highly effective, addition to the decoder that allows learnable skip-connections to encoder's feature layers, which further improves the performance.