Learning Sparse Visual Representations via Spatial-Semantic Factorization
Not on the map. Why?
Abstract (arXiv)
Self-supervised learning (SSL) faces a fundamental conflict between semantic understanding and image reconstruction. High-level semantic SSL (e.g., DINO) relies on global tokens that are forced to be location-invariant for augmentation alignment, a process that inherently discards the spatial coordinates required for reconstruction. Conversely, generative SSL (e.g., MAE) preserves dense feature grids for reconstruction but fails to produce high-level abstractions. We introduce STELLAR, a framework that resolves this tension by factorizing visual features into a low-rank product of semantic concepts and their spatial distributions. This disentanglement allows us to perform DINO-style augmentation alignment on the semantic tokens while maintaining the precise spatial mapping in the localization matrix necessary for pixel-level reconstruction. We demonstrate that as few as 16 sparse tokens under this factorized form are sufficient to simultaneously support high-quality reconstruction (2.60 FID) and match the semantic performance of dense backbones (79.10% ImageNet accuracy). Our results highlight STELLAR as a versatile sparse representation that bridges the gap between discriminative and generative vision by strategically separating semantic identity from spatial geometry. Code available at https://aka.ms/stellar.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 2 Feb 2026.
Builds on
38 of 51 references.
References · 38
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2020End-to-End Object Detection with Transformerscited by 7,384
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
26 more
- 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
- 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
- 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
- 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
- 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
- 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
- 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
- 2022MaskGIT: Masked Generative Image Transformercited by 1,082
- 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
- 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
- 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
- 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
- 2020Sparse R-CNN: End-to-End Object Detection with Learnable Proposalscited by 494
- 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
- 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
- 2016Gland Segmentation in Colon Histology Images: The GlaS Challenge Contestcited by 203
- 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
- 2018Spreading vectors for similarity searchcited by 114
- 2022SemMAE: Semantic-Guided Masking for Learning Masked Autoencoderscited by 113
- 2024Deconstructing Denoising Diffusion Models for Self-Supervised Learningcited by 89
- 2022Siamese Image Modeling for Self-Supervised Vision Representation Learningcited by 78
- 2025Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learningcited by 33
- 2024Assessing and Learning Alignment of Unimodal Vision and Language Modelscited by 31
- 2025Cluster and Predict Latent Patches for Improved Masked Image Modelingcited by 17
- 2022Improving VAE-based Representation Learningcited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.