# Investigating model performance in language identification: beyond simple error statistics

arXiv: https://arxiv.org/abs/2305.18925 · HTML: https://mlsanity.com/paper/2305.18925/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 37 references that are in the map.
- First posted on arXiv 30 May 2023.

## Builds on (most cited first, 5 of 5)

- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [Exploring wav2vec 2.0 on speaker verification and language identification](https://mlsanity.com/paper/2012.06185/) (2020)
- [Improving Language Identification of Accented Speech](https://mlsanity.com/paper/2203.16972/) (2022)
- [PHO-LID: A Unified Model Incorporating Acoustic-Phonetic and Phonotactic Information for Language Identification](https://mlsanity.com/paper/2203.12366/) (2022)

## Built on by (most cited first, 1 of 1)

- [LID Models are Actually Accent Classifiers: Implications and Solutions for LID on Accented Speech](https://mlsanity.com/paper/2506.00628/) (2025)

## Abstract (arXiv)

Language development experts need tools that can automatically identify languages from fluent, conversational speech, and provide reliable estimates of usage rates at the level of an individual recording. However, language identification systems are typically evaluated on metrics such as equal error rate and balanced accuracy, applied at the level of an entire speech corpus. These overview metrics do not provide information about model performance at the level of individual speakers, recordings, or units of speech with different linguistic characteristics. Overview statistics may therefore mask systematic errors in model performance for some subsets of the data, and consequently, have worse performance on data derived from some subsets of human speakers, creating a kind of algorithmic bias. In the current paper, we investigate how well a number of language identification systems perform on individual recordings and speech units with different linguistic properties in the MERLIon CCS Challenge. The Challenge dataset features accented English-Mandarin code-switched child-directed speech.