mlsanityData to 14 Oct 2026

Rotary Masked Autoencoders are Versatile Learners

arXiv 2505.20535 ↗First posted 26 May 2025cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Applying Transformers to irregular time-series typically requires specializations to their baseline architecture, which can result in additional computational overhead and increased method complexity. We present the Rotary Masked Autoencoder (RoMAE), which utilizes the popular Rotary Positional Embedding (RoPE) method for continuous positions. RoMAE is an extension to the Masked Autoencoder (MAE) that enables interpolation and representation learning with multidimensional continuous positional information while avoiding any time-series-specific architectural specializations. We showcase RoMAE's performance on a variety of modalities including irregular and multivariate time-series, images, and audio, demonstrating that RoMAE surpasses specialized time-series architectures on difficult datasets such as the DESC ELAsTiCC Challenge while maintaining MAE's usual performance across other modalities. In addition, we investigate RoMAE's ability to reconstruct the embedded continuous positions, demonstrating that including learned embeddings in the input sequence breaks RoPE's relative position property.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 46 of its 65 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

46 of 65 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2016Layer Normalizationcited by 5,669
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  10. 2018Neural Ordinary Differential Equationscited by 3,779
  11. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,139
34 more
  1. 2021ViViT: A Video Vision Transformercited by 1,620
  2. 2019Root Mean Square Layer Normalizationcited by 1,480
  3. 2018On the Spectral Bias of Neural Networkscited by 1,341
  4. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  5. 2016Deep Networks with Stochastic Depthcited by 979
  6. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  7. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  8. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  9. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  10. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 506
  11. 2022Transformers in Time Series: A Surveycited by 497
  12. 2020A Transformer-based Framework for Multivariate Time Series Representation Learningcited by 453
  13. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  14. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  15. 2018The UEA multivariate time series classification archive, 2018cited by 338
  16. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  17. 2022Masked Autoencoders that Listencited by 273
  18. 2021Multi-Time Attention Networks for Irregularly Sampled Time Seriescited by 238
  19. 2024Rotary Position Embedding for Vision Transformercited by 236
  20. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 208
  21. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 169
  22. 2021Modeling Irregular Time Series with Continuous Recurrent Unitscited by 103
  23. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Modelscited by 96
  24. 2024UniTS: A Unified Multi-Task Time Series Modelcited by 94
  25. 2024FiT: Flexible Vision Transformer for Diffusion Modelcited by 85
  26. 2023Time Series as Images: Vision Transformer for Irregularly Sampled Time Seriescited by 79
  27. 2022Overview frequency principle/spectral bias in deep learningcited by 69
  28. 2019Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spacescited by 67
  29. 2024ContiFormer: Continuous-Time Transformer for Irregular Time Series Modelingcited by 63
  30. 2024Base of RoPE Bounds Context Lengthcited by 52
  31. 2024RI-MAE: Rotation-Invariant Masked AutoEncoders for Self-Supervised Point Cloud Representation Learningcited by 15
  32. 2024TrackGPT -- A generative pre-trained transformer for cross-domain entity trajectory forecastingcited by 13
  33. 2024EMIT- Event-Based Masked Auto Encoding for Irregular Time Seriescited by 5
  34. 2024TrajGPT: Irregular Time-Series Representation Learning for Health Trajectory Analysiscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025StarEmbed: Benchmarking Time Series Foundation Models on Astronomical Observations of Variable Starscited by 1
  2. 2026BSAT: B-Spline Adaptive Tokenizer for Long-Term Time Series Forecastingcited by 0
  3. 2026Capturing Cardiac Cyclicity through Phase-Equivariant Self-Supervised Learningcited by 0
  4. 2026Physical Cross-Modal Masked Autoencoding for Seismic-to-Well Representation Learningcited by 0