# Spiking Neural Networks with Temporal Attention-Guided Adaptive Fusion for imbalanced Multi-modal Learning

arXiv: https://arxiv.org/abs/2505.14535 · HTML: https://mlsanity.com/paper/2505.14535/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 64 references that are in the map.
- First posted on arXiv 20 May 2025.

## Builds on (most cited first, 30 of 30)

- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting](https://mlsanity.com/paper/2202.11946/) (2022)
- [SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence](https://mlsanity.com/paper/2310.16620/) (2023)
- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks](https://mlsanity.com/paper/2202.05306/) (2022)
- [Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysis](https://mlsanity.com/paper/2406.08627/) (2024)
- [Boosting Multi-modal Model Performance with Adaptive Gradient Modulation](https://mlsanity.com/paper/2308.07686/) (2023)
- [Line-based 6-DoF Object Pose Estimation and Tracking With an Event Camera](https://mlsanity.com/paper/2408.03225/) (2024)
- [ESL-SNNs: An Evolutionary Structure Learning Strategy for Spiking Neural Networks](https://mlsanity.com/paper/2306.03693/) (2023)
- [Darwin3: A large-scale neuromorphic chip with a Novel ISA and On-Chip Learning](https://mlsanity.com/paper/2312.17582/) (2023)
- [SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition](https://mlsanity.com/paper/2308.04369/) (2023)
- [Stereo Event-based, 6-DOF Pose Tracking for Uncooperative Spacecraft](https://mlsanity.com/paper/2503.12732/) (2025)
- [Enhancing Adaptive History Reserving by Spiking Convolutional Block Attention Module in Recurrent Neural Networks](https://mlsanity.com/paper/2401.03719/) (2024)
- [Differential Coding for Training-Free ANN-to-SNN Conversion](https://mlsanity.com/paper/2503.00301/) (2025)
- [Full-DoF Egomotion Estimation for Event Cameras Using Geometric Solvers](https://mlsanity.com/paper/2503.03307/) (2025)
- [Reliable Object Tracking by Multimodal Hybrid Feature Extraction and Transformer-Based Fusion](https://mlsanity.com/paper/2405.17903/) (2024)
- [Improving the Sparse Structure Learning of Spiking Neural Networks from the View of Compression Efficiency](https://mlsanity.com/paper/2502.13572/) (2025)
- [Event-Enhanced Multi-Modal Spiking Neural Network for Dynamic Obstacle Avoidance](https://mlsanity.com/paper/2310.02361/) (2023)
- [MK-SGN: A Spiking Graph Convolutional Network with Multimodal Fusion and Knowledge Distillation for Skeleton-based Action Recognition](https://mlsanity.com/paper/2404.10210/) (2024)
- [Enhancing Audio-Visual Spiking Neural Networks through Semantic-Alignment and Cross-Modal Residual Learning](https://mlsanity.com/paper/2502.12488/) (2025)
- [Enhancing SNN-based Spatio-Temporal Learning: A Benchmark Dataset and Cross-Modality Attention Model](https://mlsanity.com/paper/2410.15689/) (2024)
- [Accurate Pose Estimation for Flight Platforms based on Divergent Multi-Aperture Imaging System](https://mlsanity.com/paper/2502.19708/) (2025)
- [Digit Recognition using Multimodal Spiking Neural Networks](https://mlsanity.com/paper/2409.00552/) (2024)
- [SNN-Driven Multimodal Human Action Recognition via Sparse Spatial-Temporal Data Fusion](https://mlsanity.com/paper/2502.13385/) (2025)
- [High-precision visual navigation device calibration method based on collimator](https://mlsanity.com/paper/2502.18012/) (2025)
- [3D Trajectory Reconstruction of Moving Points Based on a Monocular Camera](https://mlsanity.com/paper/2502.19689/) (2025)

## Built on by (most cited first, 2 of 2)

- [SETransformer: A Hybrid Attention-Based Architecture for Robust Human Activity Recognition](https://mlsanity.com/paper/2505.19369/) (2025)
- [Cross-Modal Binary Attention: An Energy-Efficient Fusion Framework for Audio-Visual Learning](https://mlsanity.com/paper/2602.00701/) (2026)

## Abstract (arXiv)

Multimodal spiking neural networks (SNNs) hold significant potential for energy-efficient sensory processing but face critical challenges in modality imbalance and temporal misalignment. Current approaches suffer from uncoordinated convergence speeds across modalities and static fusion mechanisms that ignore time-varying cross-modal interactions. We propose the temporal attention-guided adaptive fusion framework for multimodal SNNs with two synergistic innovations: 1) The Temporal Attention-guided Adaptive Fusion (TAAF) module that dynamically assigns importance scores to fused spiking features at each timestep, enabling hierarchical integration of temporally heterogeneous spike-based features; 2) The temporal adaptive balanced fusion loss that modulates learning rates per modality based on the above attention scores, preventing dominant modalities from monopolizing optimization. The proposed framework implements adaptive fusion, especially in the temporal dimension, and alleviates the modality imbalance during multimodal learning, mimicking cortical multisensory integration principles. Evaluations on CREMA-D, AVE, and EAD datasets demonstrate state-of-the-art performance (77.55%, 70.65% and 97.5%accuracy, respectively) with energy efficiency. The system resolves temporal misalignment through learnable time-warping operations and faster modality convergence coordination than baseline SNNs. This work establishes a new paradigm for temporally coherent multimodal learning in neuromorphic systems, bridging the gap between biological sensory processing and efficient machine intelligence.