# MacJEPA: Missingness-Robust Audio-Visual Recognition from Untrimmed Egocentric Videos

arXiv: https://arxiv.org/abs/2610.08192 · HTML: https://mlsanity.com/paper/2610.08192/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 40 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Builds on (most cited first, 9 of 9)

- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [Audiovisual SlowFast Networks for Video Recognition](https://mlsanity.com/paper/2001.08740/) (2020)
- [Learning and Leveraging World Models in Visual Representation Learning](https://mlsanity.com/paper/2403.00504/) (2024)
- [A-JEPA: Joint-Embedding Predictive Architecture Can Listen](https://mlsanity.com/paper/2311.15830/) (2023)
- [M&M Mix: A Multimodal Multiview Transformer Ensemble](https://mlsanity.com/paper/2206.09852/) (2022)
- [MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning](https://mlsanity.com/paper/2606.25225/) (2026)
- [Multimodal Knowledge Distillation for Egocentric Action Recognition Robust to Missing Modalities](https://mlsanity.com/paper/2504.08578/) (2025)

## Abstract (arXiv)

Audio-visual models improve egocentric action recognition by exploiting complementary cues, yet typically assume that both streams remain available at inference. Existing missing-modality methods operate on trimmed, single-event clips in which a stream is entirely present or absent, whereas real sensors fail and recover within long, untrimmed observations. We redefine egocentric modality missingness as temporally localized sensor outages within untrimmed, multi-event observations, with whole-clip absence as the limiting case. We introduce MacJEPA, a missing-modality-robust Masked-context query JEPA that recognizes visual actions and acoustic events from supplied interval queries over audio-visual context. Window-local modality dropout simulates these sensor outages during training. MacJEPA further repurposes masking in JEPA from a self-supervised pretext into a supervised robustness objective, aligning masked and clean latent representations of both multimodal content tokens and the task-conditioned queries. All objectives are optimized jointly with recognition in a single stage, requiring no test-time adaptation. Across Epic-Kitchens-100 and Epic-Sounds, a single checkpoint remains competitive under complete input and consistently surpasses published missing-modality baselines when either the dominant or auxiliary stream is removed. MacJEPA thus unifies strong full-input recognition with temporal missing-modality robustness in a single model operating on untrimmed multi-event videos.