# SpectroFusion-ViT: A Lightweight Transformer for Speech Emotion Recognition Using Harmonic Mel-Chroma Fusion

arXiv: https://arxiv.org/abs/2603.00746 · HTML: https://mlsanity.com/paper/2603.00746/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 41 references that are in the map.
- First posted on arXiv 28 Feb 2026.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention](https://mlsanity.com/paper/2305.07027/) (2023)
- [Less is More: Lighter and Faster Deep Neural Architecture for Tomato Leaf Disease Classification](https://mlsanity.com/paper/2109.02394/) (2021)
- [Fruit Quality Assessment with Densely Connected Convolutional Neural Network](https://mlsanity.com/paper/2212.04255/) (2022)
- [Rethinking Cooking State Recognition with Vision Transformers](https://mlsanity.com/paper/2212.08586/) (2022)
- [AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions](https://mlsanity.com/paper/2506.17455/) (2025)
- [HaSPeR: An Image Repository for Hand Shadow Puppet Recognition](https://mlsanity.com/paper/2408.10360/) (2024)
- [An Efficient Transfer Learning-based Approach for Apple Leaf Disease Classification](https://mlsanity.com/paper/2304.06520/) (2023)
- [MangoLeafViT: Leveraging Lightweight Vision Transformer with Runtime Augmentation for Efficient Mango Leaf Disease Classification](https://mlsanity.com/paper/2505.23961/) (2025)
- [DExNet: Combining Observations of Domain Adapted Critics for Leaf Disease Classification with Limited Data](https://mlsanity.com/paper/2506.18173/) (2025)

## Abstract (arXiv)

Speech is a natural means of conveying emotions, making it an effective method for understanding and representing human feelings. Reliable speech emotion recognition (SER) is central to applications in human-computer interaction, healthcare, education, and customer service. However, most SER methods depend on heavy backbone models or hand-crafted features that fail to balance accuracy and efficiency, particularly for low-resource languages like Bangla. In this work, we present SpectroFusion-ViT, a lightweight SER framework built utilizing EfficientViT-b0, a compact Vision Transformer architecture equipped with self-attention to capture long-range temporal and spectral patterns. The model contains only 2.04M parameters and requires 0.1 GFLOPs, enabling deployment in resource-constrained settings without compromising accuracy. Our pipeline first performs preprocessing and augmentation on raw audio, then extracts Chroma and Mel-frequency cepstral coefficient (MFCC) features. These representations are fused into a complementary time-frequency descriptor that preserves both fine-grained spectral detail and broader harmonic structure. Using transfer learning, EfficientViT-b0 is fine-tuned for multi-class emotion classification. We evaluate the system on two benchmark Bangla emotional speech datasets, SUBESCO and BanglaSER, which vary in speaker diversity, recording conditions, and acoustic characteristics. The proposed approach achieves 92.56% accuracy on SUBESCO and 82.19% on BanglaSER, surpassing existing state-of-the-art methods. These findings demonstrate that lightweight transformer architectures can deliver robust SER performance while remaining computationally efficient for real-world deployment.