Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition
Not on the map. Why?
Abstract (arXiv)
Predictive video representations do not by themselves specify how a multimodal system should handle missing sensors. We transfer frozen V-JEPA2 features to a set of heterogeneous sensors comprised by RGB video, depth video, and radar time series. A shared fusion model uses modality queries, fixed sensor slots, and explicit availability to classify breathing activity and pose from all seven non-empty sensor subsets. The contribution is a controlled evaluation of subset supervision, pooling capacity, and representation transfer on 39 OMuSense-23 subjects with five subject-group folds and three seeds. With matched downstream parameter counts, the final model reaches recording-level activity macro-F1 of 75.23%±0.60 averaged across seven subsets. Supervising all subsets in each batch improves the matched query-pooling control by 1.923 percentage points (conditional 95% subject-bootstrap interval [0.647, 3.209]), while query pooling shows no clear advantage over matched mean pooling under balanced supervision. Radar probes support transfer beyond pooled wavelet pixels, 64-token inference approximately preserves mean activity performance, and the tested Consensus-JEPA loss adds no measured benefit against its exact control.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 18 references that are in the map. First posted on arXiv 11 Oct 2026.
Builds on
14 of 18 references.
References · 14
- 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
- 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
- 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
- 2022Are Multimodal Transformers Robust to Missing Modality?cited by 186
- 2023Multi-modal Learning with Missing Modality via Shared-Specific Feature Modellingcited by 163
- 2022Towards Good Practices for Missing Modality Robust Action Recognitioncited by 54
- 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 29
- 2025SimMLM: A Simple Framework for Multi-modal Learning with Missing Modalitycited by 23
- 2024M3-JEPA: Multimodal Alignment via Multi-gate MoE based on the Joint-Embedding Predictive Architecturecited by 13
- 2024Time-Series JEPA for Predictive Remote Control under Capacity-Limited Networkscited by 9
- 2020Input Dropout for Spatially Aligned Modalitiescited by 5
Built on by
Not cited yet.