mlsanityData to 7 Oct 2026

MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis

arXiv 2211.09117 ↗First posted 16 Nov 2022cs.CVcited by 222

Not on the map. Why?

Cited by 44 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generative modeling and representation learning are two key tasks in computer vision. However, these models are typically trained independently, which ignores the potential for each task to help the other, and leads to training and model maintenance overheads. In this work, we propose MAsked Generative Encoder (MAGE), the first framework to unify SOTA image generation and self-supervised representation learning. Our key insight is that using variable masking ratios in masked image modeling pre-training can allow generative training (very high masking ratio) and representation learning (lower masking ratio) under the same training framework. Inspired by previous generative models, MAGE uses semantic tokens learned by a vector-quantized GAN at inputs and outputs, combining this with masking. We can further improve the representation by adding a contrastive loss to the encoder output. We extensively evaluate the generation and representation learning capabilities of MAGE. On ImageNet-1K, a single MAGE ViT-L model obtains 9.10 FID in the task of class-unconditional image generation and 78.9% top-1 accuracy for linear probing, achieving state-of-the-art performance in both image generation and representation learning. Code is available at https://github.com/LTH14/mage.

Key facts. Cited by 222 papers in the map (data through 7 Oct 2026). Builds on 61 of its 68 references that are in the map. First posted on arXiv 16 Nov 2022.

Builds on

61 of 68 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2016Improved Techniques for Training GANscited by 2,938
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  13. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  14. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  15. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  16. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  17. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  18. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  19. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  20. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  21. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  22. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  23. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  24. 2019Contrastive Multiview Codingcited by 1,297
  25. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  26. 2016Deep Networks with Stochastic Depthcited by 973
  27. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  28. 2018Self-Attention Generative Adversarial Networkscited by 809
  29. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  30. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  31. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  32. 2017Large Batch Training of Convolutional Networkscited by 529
  33. 2016Adversarial Feature Learningcited by 350
  34. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  35. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  36. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  37. 2021Generating Images with Sparse Representationscited by 262
  38. 2019Large Scale Adversarial Representation Learningcited by 238

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 36; 2024: 64; 2025: 78; 2026: 44.

222 papers.

Sort:

By area:

  1. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  2. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  3. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  4. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  5. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
  6. 2023Fast Training of Diffusion Models with Masked Transformerscited by 172
  7. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  8. 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
  9. 20234M: Massively Multimodal Masked Modelingcited by 118
  10. 2024OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generationcited by 117
  11. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  12. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
38 more
  1. 2023Return of Unconditional Generation: A Self-supervised Representation Generation Methodcited by 95
  2. 2025EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modelingcited by 85
  3. 2024RandAR: Decoder-only Autoregressive Visual Generation in Random Orderscited by 82
  4. 2023GIVT: Generative Infinite-Vocabulary Transformerscited by 80
  5. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  6. 2024Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%cited by 68
  7. 2023Diffusion Models Beat GANs on Image Classificationcited by 57
  8. 2024SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizercited by 55
  9. 2024Diffusion Models and Representation Learning: A Surveycited by 53
  10. 20244M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalitiescited by 52
  11. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scalecited by 49
  12. 2025Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokenscited by 48
  13. 2024Machine Unlearning for Image-to-Image Generative Modelscited by 47
  14. 2024SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformercited by 47
  15. 2024ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformercited by 45
  16. 2024Scalable Image Tokenization with Index Backpropagation Quantizationcited by 44
  17. 2025Fractal Generative Modelscited by 44
  18. 2024VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modelingcited by 42
  19. 2024Autoregressive Models in Vision: A Surveycited by 39
  20. 2024TokenRec: Learning to Tokenize ID for LLM-based Generative Recommendationcited by 35
  21. 2023Do text-free diffusion models learn discriminative visual representations?cited by 33
  22. 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 32
  23. 2024WildFake: A Large-scale Challenging Dataset for AI-Generated Images Detectioncited by 31
  24. 2025Halton Scheduler For Masked Generative Image Transformercited by 30
  25. 2024Rethinking Patch Dependence for Masked Autoencoderscited by 29
  26. 2024Masked Image Modeling: A Surveycited by 28
  27. 2024Cross-Modal Conditioned Reconstruction for Language-guided Medical Image Segmentationcited by 27
  28. 2023A Pytorch Reproduction of Masked Generative Image Transformercited by 25
  29. 2025USP: Unified Self-Supervised Pretraining for Image Generation and Understandingcited by 24
  30. 2024From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generationcited by 23
  31. 2024Revisiting Non-Autoregressive Transformers for Efficient Image Synthesiscited by 22
  32. 2024Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspectivecited by 22
  33. 2024Customize Your Visual Autoregressive Recipe with Set Autoregressive Modelingcited by 21
  34. 2024FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learningcited by 21
  35. 2025Next-Embedding Prediction Makes Strong Vision Learnerscited by 21
  36. 2025The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencodingcited by 21
  37. 2023ADDP: Learning General Representations for Image Recognition and Generation with Alternating Denoising Diffusion Processcited by 19
  38. 2025Autoregressive Image Generation with Randomized Parallel Decodingcited by 19

50 most cited of 222 All 222 (JSON)