# Transformer Redesign for Late Fusion of Audio-Text Features on Ultra-Low-Power Edge Hardware

arXiv: https://arxiv.org/abs/2510.18036 · HTML: https://mlsanity.com/paper/2510.18036/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 48 references that are in the map.
- First posted on arXiv 20 Oct 2025.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [Hello Edge: Keyword Spotting on Microcontrollers](https://mlsanity.com/paper/1711.07128/) (2017)
- [Deep Residual Learning for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1710.10361/) (2017)
- [Speech Emotion Recognition using Self-Supervised Features](https://mlsanity.com/paper/2202.03896/) (2022)
- [Temporal Convolution for Real-time Keyword Spotting on Mobile Devices](https://mlsanity.com/paper/1904.03814/) (2019)
- [Broadcasted Residual Learning for Efficient Keyword Spotting](https://mlsanity.com/paper/2106.04140/) (2021)
- [An Evaluation of Edge TPU Accelerators for Convolutional Neural Networks](https://mlsanity.com/paper/2102.10423/) (2021)
- [Multimodal Emotion Recognition with High-level Speech and Text Features](https://mlsanity.com/paper/2111.10202/) (2021)
- [Depthwise Separable Convolutional ResNet with Squeeze-and-Excitation Blocks for Small-footprint Keyword Spotting](https://mlsanity.com/paper/2004.12200/) (2020)
- [Small-Footprint Keyword Spotting with Multi-Scale Temporal Convolution](https://mlsanity.com/paper/2010.09960/) (2020)
- [Speech Emotion Recognition with Multiscale Area Attention and Data Augmentation](https://mlsanity.com/paper/2102.01813/) (2021)
- [Small-footprint Keyword Spotting with Graph Convolutional Network](https://mlsanity.com/paper/1912.05124/) (2019)
- [Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models](https://mlsanity.com/paper/2202.08974/) (2022)
- [Privacy-preserving Speech Emotion Recognition through Semi-Supervised Federated Learning](https://mlsanity.com/paper/2202.02611/) (2022)
- [Improving vision-inspired keyword spotting using dynamic module skipping in streaming conformer encoder](https://mlsanity.com/paper/2309.00140/) (2023)
- [Real-time Speech Emotion Recognition Based on Syllable-Level Feature Extraction](https://mlsanity.com/paper/2204.11382/) (2022)
- [Privacy Enhanced Speech Emotion Communication using Deep Learning Aided Edge Computing](https://mlsanity.com/paper/2103.17139/) (2021)
- [Emotions Beyond Words: Non-Speech Audio Emotion Recognition With Edge Computing](https://mlsanity.com/paper/2305.00725/) (2023)

## Abstract (arXiv)

Deploying emotion recognition systems in real-world environments where devices must be small, low-power, and private remains a significant challenge. This is especially relevant for applications such as tension monitoring, conflict de-escalation, and responsive wearables, where cloud-based solutions are impractical. Multimodal emotion recognition has advanced through deep learning, but most systems remain unsuitable for deployment on ultra-constrained edge devices. Prior work typically relies on powerful hardware, lacks real-time performance, or uses unimodal input. This paper addresses that gap by presenting a hardware-aware emotion recognition system that combines acoustic and linguistic features using a late-fusion architecture optimised for Edge TPU. The design integrates a quantised transformer-based acoustic model with frozen keyword embeddings from a DSResNet-SE network, enabling real-time inference within a 1.8MB memory budget and 21-23ms latency. The pipeline ensures spectrogram alignment between training and deployment using MicroFrontend and MLTK. Evaluation on re-recorded, segmented IEMOCAP samples captured through the Coral Dev Board Micro microphone shows a 6.3% macro F1 improvement over unimodal baselines. This work demonstrates that accurate, real-time multimodal emotion inference is achievable on microcontroller-class edge platforms through task-specific fusion and hardware-guided model design.