# Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals

arXiv: https://arxiv.org/abs/2309.05927 · HTML: https://mlsanity.com/paper/2309.05927/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 69 references that are in the map.
- First posted on arXiv 12 Sep 2023.

## Builds on (most cited first, 30 of 43)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks](https://mlsanity.com/paper/2206.08916/) (2022)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles](https://mlsanity.com/paper/2306.00989/) (2023)
- [Global Filter Networks for Image Classification](https://mlsanity.com/paper/2107.00645/) (2021)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers](https://mlsanity.com/paper/2111.13587/) (2021)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients](https://mlsanity.com/paper/2005.13249/) (2020)
- [DeepSleepNet: a Model for Automatic Sleep Stage Scoring based on Raw Single-Channel EEG](https://mlsanity.com/paper/1703.04046/) (2017)
- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [Self-Supervised Representation Learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2110.09327/) (2021)
- [FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting](https://mlsanity.com/paper/2205.08897/) (2022)
- [SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling](https://mlsanity.com/paper/2302.00861/) (2023)

## Built on by (most cited first, 23 of 23)

- [Wearable Accelerometer Foundation Models for Health via Knowledge Distillation](https://mlsanity.com/paper/2412.11276/) (2024)
- [DetailCLIP: Detail-Oriented CLIP for Fine-Grained Tasks](https://mlsanity.com/paper/2409.06809/) (2024)
- [Promoting cross-modal representations to improve multimodal foundation models for physiological signals](https://mlsanity.com/paper/2410.16424/) (2024)
- [HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling with Self-Distillation for Long-Term Forecasting](https://mlsanity.com/paper/2401.05012/) (2024)
- [FLEXtime: Filterbank learning to explain time series](https://mlsanity.com/paper/2411.05841/) (2024)
- [Wavelet-Driven Masked Multiscale Reconstruction for PPG Foundation Models](https://mlsanity.com/paper/2601.12215/) (2026)
- [Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning](https://mlsanity.com/paper/2409.10362/) (2024)
- [Wearable Foundation Models Should Go Beyond Static Encoders](https://mlsanity.com/paper/2603.19564/) (2026)
- [A robust PPG foundation model using multimodal physiological supervision](https://mlsanity.com/paper/2606.07365/) (2026)
- [Balanced Data, Imbalanced Spectra: Unveiling Class Disparities with Spectral Imbalance](https://mlsanity.com/paper/2402.11742/) (2024)
- [Few-Shot Radar Signal Recognition through Self-Supervised Learning and Radio Frequency Domain Adaptation](https://mlsanity.com/paper/2501.03461/) (2025)
- [Learning Clinical Representations Under Systematic Distribution Shift](https://mlsanity.com/paper/2603.07348/) (2026)
- [Discriminative Representation Learning for Clinical Prediction](https://mlsanity.com/paper/2603.20921/) (2026)
- [Generalizable autoregressive modeling of time series through functional narratives](https://mlsanity.com/paper/2410.08421/) (2024)
- [Structured-Noise Masked Modeling for Video, Audio and Beyond](https://mlsanity.com/paper/2503.16311/) (2025)
- [BioX-Bridge: Model Bridging for Unsupervised Cross-Modal Knowledge Transfer across Biosignals](https://mlsanity.com/paper/2510.02276/) (2025)
- [Learning Time-Scale Invariant Population-Level Neural Representations](https://mlsanity.com/paper/2511.13022/) (2025)
- [Frequency-Aware Masked Autoencoders for Human Activity Recognition using Accelerometers](https://mlsanity.com/paper/2502.17477/) (2025)
- [PULSE: Privileged Knowledge Transfer from Rich to Deployable Sensors for Embodied Multi-Sensory Learning](https://mlsanity.com/paper/2510.24058/) (2025)
- [Event Fields: Learning Latent Event Structure for Waveform Foundation Models](https://mlsanity.com/paper/2605.08685/) (2026)
- [PAMF: Prior-Aware Multimodal Fusion for Incomplete Time Series Data](https://mlsanity.com/paper/2606.06328/) (2026)
- [Neonatal Hypoxic-ischaemic Encephalopathy Classification from the EEG and HRV Signals Using a Conformer based Masked Autoencoder](https://mlsanity.com/paper/2607.23554/) (2026)
- [LiteEMG-FM: An Efficient and Deployable Foundation Model for Robust EMG Sensing](https://mlsanity.com/paper/2610.02497/) (2026)

## Abstract (arXiv)

Leveraging multimodal information from biosignals is vital for building a comprehensive representation of people's physical and mental states. However, multimodal biosignals often exhibit substantial distributional shifts between pretraining and inference datasets, stemming from changes in task specification or variations in modality compositions. To achieve effective pretraining in the presence of potential distributional shifts, we propose a frequency-aware masked autoencoder (textttbioFAME) that learns to parameterize the representation of biosignals in the frequency space. textttbioFAME incorporates a frequency-aware transformer, which leverages a fixed-size Fourier-based operator for global token mixing, independent of the length and sampling rate of inputs. To maintain the frequency components within each input channel, we further employ a frequency-maintain pretraining strategy that performs masked autoencoding in the latent space. The resulting architecture effectively utilizes multimodal information during pretraining, and can be seamlessly adapted to diverse tasks and modalities at test time, regardless of input size and order. We evaluated our approach on a diverse set of transfer experiments on unimodal time series, achieving an average of uparrow5.5% improvement in classification accuracy over the previous state-of-the-art. Furthermore, we demonstrated that our architecture is robust in modality mismatch scenarios, including unpredicted modality dropout or substitution, proving its practical utility in real-world applications. Code is available at https://github.com/apple/ml-famae .