# LSTMSE-Net: Long Short Term Speech Enhancement Network for Audio-visual Speech Enhancement

arXiv: https://arxiv.org/abs/2409.02266 · HTML: https://mlsanity.com/paper/2409.02266/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 27 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results](https://mlsanity.com/paper/2005.13981/) (2020)
- [The Conversation: Deep Audio-Visual Speech Enhancement](https://mlsanity.com/paper/1804.04121/) (2018)
- [An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation](https://mlsanity.com/paper/2008.09586/) (2020)
- [Phase-aware Speech Enhancement with Deep Complex U-Net](https://mlsanity.com/paper/1903.03107/) (2019)
- [Discriminative Multi-modality Speech Recognition](https://mlsanity.com/paper/2005.05592/) (2020)
- [Improved Lite Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2008.13222/) (2020)
- [Audio-Visual Speech Enhancement and Separation by Utilizing Multi-Modal Self-Supervised Embeddings](https://mlsanity.com/paper/2210.17456/) (2022)
- [Leveraging Self-Supervised Audio-Visual Pretrained Models to Improve Vocoded Speech Intelligibility in Cochlear Implant Simulation](https://mlsanity.com/paper/2307.07748/) (2023)

## Built on by (most cited first, 3 of 3)

- [Bridging The Multi-Modality Gaps of Audio, Visual and Linguistic for Speech Enhancement](https://mlsanity.com/paper/2501.13375/) (2025)
- [AUREXA-SE: Audio-Visual Unified Representation Exchange Architecture with Cross-Attention and Squeezeformer for Speech Enhancement](https://mlsanity.com/paper/2510.05295/) (2025)
- [Lightweight Wasserstein Audio-Visual Model for Unified Speech Enhancement and Separation](https://mlsanity.com/paper/2512.06689/) (2025)

## Abstract (arXiv)

In this paper, we propose long short term memory speech enhancement network (LSTMSE-Net), an audio-visual speech enhancement (AVSE) method. This innovative method leverages the complementary nature of visual and audio information to boost the quality of speech signals. Visual features are extracted with VisualFeatNet (VFN), and audio features are processed through an encoder and decoder. The system scales and concatenates visual and audio features, then processes them through a separator network for optimized speech enhancement. The architecture highlights advancements in leveraging multi-modal data and interpolation techniques for robust AVSE challenge systems. The performance of LSTMSE-Net surpasses that of the baseline model from the COG-MHEAR AVSE Challenge 2024 by a margin of 0.06 in scale-invariant signal-to-distortion ratio (SISDR), 0.03 in short-time objective intelligibility (STOI), and 1.32 in perceptual evaluation of speech quality (PESQ). The source code of the proposed LSTMSE-Net is available at urlhttps://github.com/mtanveer1/AVSEC-3-Challenge.