# Cross-Temporal Attention Fusion (CTAF) for Multimodal Physiological Signals in Self-Supervised Learning

arXiv: https://arxiv.org/abs/2602.02784 · HTML: https://mlsanity.com/paper/2602.02784/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [Soft-DTW: a Differentiable Loss Function for Time-Series](https://mlsanity.com/paper/1703.01541/) (2017)
- [Learning Correspondence from the Cycle-Consistency of Time](https://mlsanity.com/paper/1903.07593/) (2019)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients](https://mlsanity.com/paper/2005.13249/) (2020)
- [Temporal Cycle-Consistency Learning](https://mlsanity.com/paper/1904.07846/) (2019)
- [K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations](https://mlsanity.com/paper/2005.04120/) (2020)
- [EEG Foundation Challenge: From Cross-Task to Cross-Subject EEG Decoding](https://mlsanity.com/paper/2506.19141/) (2025)
- [Hypercomplex Multimodal Emotion Recognition from EEG and Peripheral Physiological Signals](https://mlsanity.com/paper/2310.07648/) (2023)
- [Hierarchical Hypercomplex Network for Multimodal Emotion Recognition](https://mlsanity.com/paper/2409.09194/) (2024)

## Built on by (most cited first, 1 of 1)

- [BioSync: Transformer-Based Cross-Modal Fusion for a Multimodal Physiological Digital Biomarker](https://mlsanity.com/paper/2609.04504/) (2026)

## Abstract (arXiv)

We study multimodal affect modeling when EEG and peripheral physiology are asynchronous, which most fusion methods ignore or handle with costly warping. We propose Cross-Temporal Attention Fusion (CTAF), a self-supervised module that learns soft bidirectional alignments between modalities and builds a robust clip embedding using time-aware cross attention, a lightweight fusion gate, and alignment-regularized contrastive objectives with optional weak supervision. On the K-EmoCon dataset, under leave-one-out cross-validation evaluation, CTAF yields higher cosine margins for matched pairs and better cross-modal token retrieval within one second, and it is competitive with the baseline on three-bin accuracy and macro-F1 while using few labels. Our contributions are a time-aware fusion mechanism that directly models correspondence, an alignment-driven self-supervised objective tailored to EEG and physiology, and an evaluation protocol that measures alignment quality itself. Our approach accounts for the coupling between the central and autonomic nervous systems in psychophysiological time series. These results indicate that CTAF is a strong step toward label-efficient, generalizable EEG-peripheral fusion under temporal asynchrony.