# Acoustic Feature Mixup for Balanced Multi-aspect Pronunciation Assessment

arXiv: https://arxiv.org/abs/2406.15723 · HTML: https://mlsanity.com/paper/2406.15723/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 22 Jun 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup](https://mlsanity.com/paper/2009.06962/) (2020)
- [SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization](https://mlsanity.com/paper/2006.01791/) (2020)
- [Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity](https://mlsanity.com/paper/2102.03065/) (2021)
- [speechocean762: An Open-Source Non-native English Speech Corpus For Pronunciation Assessment](https://mlsanity.com/paper/2104.01378/) (2021)
- [TokenMix: Rethinking Image Mixing for Data Augmentation in Vision Transformers](https://mlsanity.com/paper/2207.08409/) (2022)
- [Transformer-Based Multi-Aspect Multi-Granularity Non-Native English Speaker Pronunciation Assessment](https://mlsanity.com/paper/2205.03432/) (2022)
- [Context-aware Goodness of Pronunciation for Computer-Assisted Pronunciation Training](https://mlsanity.com/paper/2008.08647/) (2020)
- [3M: An Effective Multi-view, Multi-granularity, and Multi-aspect Modeling Approach to English Pronunciation Assessment](https://mlsanity.com/paper/2208.09110/) (2022)
- [Harnessing Hard Mixed Samples with Decoupled Regularizer](https://mlsanity.com/paper/2203.10761/) (2022)
- [Hierarchical Pronunciation Assessment with Multi-Aspect Attention](https://mlsanity.com/paper/2211.08102/) (2022)
- [A Hierarchical Context-aware Modeling Approach for Multi-aspect and Multi-granular Pronunciation Assessment](https://mlsanity.com/paper/2305.18146/) (2023)
- [A transfer learning based approach for pronunciation scoring](https://mlsanity.com/paper/2111.00976/) (2021)
- [Score-balanced Loss for Multi-aspect Pronunciation Assessment](https://mlsanity.com/paper/2305.16664/) (2023)
- [Embedding Space Interpolation Beyond Mini-Batch, Beyond Pairs and Beyond Examples](https://mlsanity.com/paper/2311.05538/) (2023)

## Built on by (most cited first, 3 of 3)

- [Evaluating Logit-Based GOP Scores for Mispronunciation Detection](https://mlsanity.com/paper/2506.12067/) (2025)
- [Multi-task Pretraining for Enhancing Interpretable L2 Pronunciation Assessment](https://mlsanity.com/paper/2509.16876/) (2025)
- [When Speech Meets Lips: Interpretable Audio-Visual Synchronization for L2 Pronunciation Assessment](https://mlsanity.com/paper/2609.06788/) (2026)

## Abstract (arXiv)

In automated pronunciation assessment, recent emphasis progressively lies on evaluating multiple aspects to provide enriched feedback. However, acquiring multi-aspect-score labeled data for non-native language learners' speech poses challenges; moreover, it often leads to score-imbalanced distributions. In this paper, we propose two Acoustic Feature Mixup strategies, linearly and non-linearly interpolating with the in-batch averaged feature, to address data scarcity and score-label imbalances. Primarily using goodness-of-pronunciation as an acoustic feature, we tailor mixup designs to suit pronunciation assessment. Further, we integrate fine-grained error-rate features by comparing speech recognition results with the original answer phonemes, giving direct hints for mispronunciation. Effective mixing of the acoustic features notably enhances overall scoring performances on the speechocean762 dataset, and detailed analysis highlights our potential to predict unseen distortions.