# BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognition

arXiv: https://arxiv.org/abs/1907.00457 · HTML: https://mlsanity.com/paper/1907.00457/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jun 2019.

## Built on by (most cited first, 10 of 10)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://mlsanity.com/paper/2101.07597/) (2021)
- [Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction](https://mlsanity.com/paper/2110.15430/) (2021)
- [ASR-Generated Text for Language Model Pre-training Applied to Speech Tasks](https://mlsanity.com/paper/2207.01893/) (2022)
- [Accidental Learners: Spoken Language Identification in Multilingual Self-Supervised Models](https://mlsanity.com/paper/2211.05103/) (2022)
- [Improving Pseudo-label Training For End-to-end Speech Recognition Using Gradient Mask](https://mlsanity.com/paper/2110.04056/) (2021)
- [Similarity and Content-based Phonetic Self Attention for Speech Recognition](https://mlsanity.com/paper/2203.10252/) (2022)
- [Using Pause Information for More Accurate Entity Recognition](https://mlsanity.com/paper/2109.13222/) (2021)
- [Enhance Language Identification using Dual-mode Model with Knowledge Distillation](https://mlsanity.com/paper/2203.03218/) (2022)
- [Guided contrastive self-supervised pre-training for automatic speech recognition](https://mlsanity.com/paper/2210.12335/) (2022)

## Abstract (arXiv)

We introduce BERTphone, a Transformer encoder trained on large speech corpora that outputs phonetically-aware contextual representation vectors that can be used for both speaker and language recognition. This is accomplished by training on two objectives: the first, inspired by adapting BERT to the continuous domain, involves masking spans of input frames and reconstructing the whole sequence for acoustic representation learning; the second, inspired by the success of bottleneck features from ASR, is a sequence-level CTC loss applied to phoneme labels for phonetic representation learning. We pretrain two BERTphone models (one on Fisher and one on TED-LIUM) and use them as feature extractors into x-vector-style DNNs for both tasks. We attain a state-of-the-art Cavg of 6.16 on the challenging LRE07 3sec closed-set language recognition task. On Fisher and VoxCeleb speaker recognition tasks, we see an 18% relative reduction in speaker EER when training on BERTphone vectors instead of MFCCs. In general, BERTphone outperforms previous phonetic pretraining approaches on the same data. We release our code and models at https://github.com/awslabs/speech-representations.