# VANPY: Voice Analysis Framework

arXiv: https://arxiv.org/abs/2502.17579 · HTML: https://mlsanity.com/paper/2502.17579/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 99 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 24 of 24)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Attention is All You Need in Speech Separation](https://mlsanity.com/paper/2010.13154/) (2020)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement](https://mlsanity.com/paper/2104.03538/) (2021)
- [End-to-end speaker segmentation for overlap-aware resegmentation](https://mlsanity.com/paper/2104.04045/) (2021)
- [Speaker Diarization with LSTM](https://mlsanity.com/paper/1710.10468/) (2017)
- [Speaker Recognition Based on Deep Learning: An Overview](https://mlsanity.com/paper/2012.00931/) (2020)
- [ECAPA-TDNN Embeddings for Speaker Diarization](https://mlsanity.com/paper/2104.01466/) (2021)
- [Survey on Deep Neural Networks in Speech and Vision Systems](https://mlsanity.com/paper/1908.07656/) (2019)
- [VoxCeleb Enrichment for Age and Gender Recognition](https://mlsanity.com/paper/2109.13510/) (2021)
- [Speech-based Age and Gender Prediction with Transformers](https://mlsanity.com/paper/2306.16962/) (2023)
- [Speaker Naming in Movies](https://mlsanity.com/paper/1809.08761/) (2018)
- [Convolutional Neural Networks and x-vector Embedding for DCASE2018 Acoustic Scene Classification Challenge](https://mlsanity.com/paper/1810.04273/) (2018)
- [End-to-End Speaker Height and age estimation using Attention Mechanism with LSTM-RNN](https://mlsanity.com/paper/2101.05056/) (2021)
- [Towards the Objective Speech Assessment of Smoking Status based on Voice Features: A Review of the Literature](https://mlsanity.com/paper/2106.07874/) (2021)

## Abstract (arXiv)

Voice data is increasingly being used in modern digital communications, yet there is still a lack of comprehensive tools for automated voice analysis and characterization. To this end, we developed the VANPY (Voice Analysis in Python) framework for automated pre-processing, feature extraction, and classification of voice data. The VANPY is an open-source end-to-end comprehensive framework that was developed for the purpose of speaker characterization from voice data. The framework is designed with extensibility in mind, allowing for easy integration of new components and adaptation to various voice analysis applications. It currently incorporates over fifteen voice analysis components - including music/speech separation, voice activity detection, speaker embedding, vocal feature extraction, and various classification models. Four of the VANPY's components were developed in-house and integrated into the framework to extend its speaker characterization capabilities: gender classification, emotion classification, age regression, and height regression. The models demonstrate robust performance across various datasets, although not surpassing state-of-the-art performance. As a proof of concept, we demonstrate the framework's ability to extract speaker characteristics on a use-case challenge of analyzing character voices from the movie "Pulp Fiction." The results illustrate the framework's capability to extract multiple speaker characteristics, including gender, age, height, emotion type, and emotion intensity measured across three dimensions: arousal, dominance, and valence.