mlsanityData to 7 Oct 2026

Gaussian Masked Autoencoders

arXiv 2501.03229 ↗First posted 6 Jan 2025cs.CV cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

This paper explores Masked Autoencoders (MAE) with Gaussian Splatting. While reconstructive self-supervised learning frameworks such as MAE learns good semantic abstractions, it is not trained for explicit spatial awareness. Our approach, named Gaussian Masked Autoencoder, or GMAE, aims to learn semantic abstractions and spatial understanding jointly. Like MAE, it reconstructs the image end-to-end in the pixel space, but beyond MAE, it also introduces an intermediate, 3D Gaussian-based representation and renders images via splatting. We show that GMAE can enable various zero-shot learning capabilities of spatial understanding (e.g., figure-ground segmentation, image layering, edge detection, etc.) while preserving the high-level semantics of self-supervised representation quality from MAE. To our knowledge, we are the first to employ Gaussian primitives in an image representation learning framework beyond optimization-based single-scene reconstructions. We believe GMAE will inspire further research in this direction and contribute to developing next-generation techniques for modeling high-fidelity visual data. More details at https://brjathu.github.io/gmae

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 6 Jan 2025.

Builds on

32 of 42 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
20 more
  1. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  7. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  8. 2020Object-Centric Learning with Slot Attentioncited by 865
  9. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  10. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  11. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  12. 2017Large Batch Training of Convolutional Networkscited by 529
  13. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
  14. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  15. 2020Single-View View Synthesis with Multiplane Imagescited by 263
  16. 2022Visual Prompting via Image Inpaintingcited by 248
  17. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  18. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  19. 2022Self-Supervised Learning of Object Parts for Semantic Segmentationcited by 90
  20. 2018Superpixel Sampling Networkscited by 79

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Can Video Diffusion Model Reconstruct 4D Geometry?cited by 10
  2. 2025GViT: Representing Images as Gaussians for Visual Recognitioncited by 1
  3. 2025Tracking by Predicting 3-D Gaussians Over Timecited by 0
  4. 2026PADFormer: Pose-agnostic Anomaly Detection from Sparse View Imagescited by 0
  5. 2026Masked Swingers: Harnessing Data Augmentation to Advance Autoencoders for Self-Supervised Learningcited by 0