# DeepMSRF: A novel Deep Multimodal Speaker Recognition framework with Feature selection

arXiv: https://arxiv.org/abs/2007.06809 · HTML: https://mlsanity.com/paper/2007.06809/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2020.

## Built on by (most cited first, 8 of 8)

- [A Concise Review of Transfer Learning](https://mlsanity.com/paper/2104.02144/) (2021)
- [Embodied AI-Driven Operation of Smart Cities: A Concise Review](https://mlsanity.com/paper/2108.09823/) (2021)
- [DRDrV3: Complete Lesion Detection in Fundus Images Using Mask R-CNN, Transfer Learning, and LSTM](https://mlsanity.com/paper/2108.08095/) (2021)
- [Data Analytics for Smart cities: Challenges and Promises](https://mlsanity.com/paper/2109.05581/) (2021)
- [MVSA-Net: Multi-View State-Action Recognition for Robust and Deployable Trajectory Generation](https://mlsanity.com/paper/2311.08393/) (2023)
- [Deep Learning in Healthcare: An In-Depth Analysis](https://mlsanity.com/paper/2302.10904/) (2023)
- [Applications of Machine Learning in Healthcare and Internet of Things (IOT): A Comprehensive Review](https://mlsanity.com/paper/2202.02868/) (2022)
- [Decoding the Alphabet Soup of Degrees in the United States Postsecondary Education System Through Hybrid Method: Database and Text Mining](https://mlsanity.com/paper/2309.13050/) (2023)

## Abstract (arXiv)

For recognizing speakers in video streams, significant research studies have been made to obtain a rich machine learning model by extracting high-level speaker's features such as facial expression, emotion, and gender. However, generating such a model is not feasible by using only single modality feature extractors that exploit either audio signals or image frames, extracted from video streams. In this paper, we address this problem from a different perspective and propose an unprecedented multimodality data fusion framework called DeepMSRF, Deep Multimodal Speaker Recognition with Feature selection. We execute DeepMSRF by feeding features of the two modalities, namely speakers' audios and face images. DeepMSRF uses a two-stream VGGNET to train on both modalities to reach a comprehensive model capable of accurately recognizing the speaker's identity. We apply DeepMSRF on a subset of VoxCeleb2 dataset with its metadata merged with VGGFace2 dataset. The goal of DeepMSRF is to identify the gender of the speaker first, and further to recognize his or her name for any given video stream. The experimental results illustrate that DeepMSRF outperforms single modality speaker recognition methods with at least 3 percent accuracy.