mlsanityData to 7 Oct 2026

Rejuvenating image-GPT as Strong Visual Representation Learners

arXiv 2312.02147 ↗First posted 4 Dec 2023cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

This paper enhances image-GPT (iGPT), one of the pioneering works that introduce autoregressive pretraining to predict the next pixels for visual representation learning. Two simple yet essential changes are made. First, we shift the prediction target from raw pixels to semantic tokens, enabling a higher-level understanding of visual content. Second, we supplement the autoregressive modeling by instructing the model to predict not only the next tokens but also the visible tokens. This pipeline is particularly effective when semantic tokens are encoded by discriminatively trained models, such as CLIP. We introduce this novel approach as D-iGPT. Extensive experiments showcase that D-iGPT excels as a strong learner of visual representations: A notable achievement is its compelling performance on the ImageNet-1K dataset -- by training on publicly available datasets, D-iGPT unprecedentedly achieves 90.0% top-1 accuracy with a vanilla ViT-H. Additionally, D-iGPT shows strong generalization on the downstream task. Code is available at https://github.com/OliverRensu/D-iGPT.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 59 of its 67 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

59 of 67 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  10. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  10. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  13. 2019Natural Adversarial Examplescited by 1,401
  14. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  15. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  16. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  17. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  18. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  19. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  20. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  21. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  22. 2021Scaling Vision Transformerscited by 993
  23. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  24. 2016Pixel Recurrent Neural Networkscited by 814
  25. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  26. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  27. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  28. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  29. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  30. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  31. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  32. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  33. 2020Are we done with ImageNet?cited by 329
  34. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  35. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  36. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  37. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  38. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 5; 2026: 5.

16 papers.

Sort:

By area:

  1. 2024Dinomaly: The Less Is More Philosophy in Multi-Class Unsupervised Anomaly Detectioncited by 92
  2. 2024FlowAR: Scale-wise Autoregressive Image Generation Meets Flow Matchingcited by 77
  3. 2024UltraPixel: Advancing Ultra-High-Resolution Image Synthesis to New Peakscited by 64
  4. 2024Autoregressive Pretraining with Mamba in Visioncited by 21
  5. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  6. 2025Dinomaly2: A Unified Framework for Unsupervised Image Anomaly Detectioncited by 6
  7. 2025Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction?cited by 2
  8. 2026Is Task-Specific Training Necessary for Anomaly Detection?cited by 2
  9. 2026OmniLottie: Generating Vector Animations via Parameterized Lottie Tokenscited by 2
  10. 2025Search is All You Need for Few-shot Anomaly Detectioncited by 1
  11. 2025Deciphering Personalization: Towards Fine-Grained Explainability in Natural Language for Personalized Image Generation Modelscited by 1
  12. 2024MAL: Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performancecited by 0
4 more
  1. 2025MixAR: Mixture Autoregressive Image Generationcited by 0
  2. 2026Performance of Small Language Model Pretraining on FABRIC: An Empirical Studycited by 0
  3. 2026Separators in Enhancing Autoregressive Pretraining for Vision Mambacited by 0
  4. 2026JASPR: Joint Spatial Representation learning of histology and spatial genomics for improved virtual genomic screening and clinical prognosticationcited by 0