# Speaker-Conditioned Hierarchical Modeling for Automated Speech Scoring

arXiv: https://arxiv.org/abs/2109.00928 · HTML: https://mlsanity.com/paper/2109.00928/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 76 references that are in the map.
- First posted on arXiv 30 Aug 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Not Just a Black Box: Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1605.01713/) (2016)
- [Hierarchical Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1810.07217/) (2018)
- [Did the Model Understand the Question?](https://mlsanity.com/paper/1805.05492/) (2018)
- [What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure](https://mlsanity.com/paper/2101.00387/) (2021)
- [Learning Private Neural Language Modeling with Attentive Aggregation](https://mlsanity.com/paper/1812.07108/) (2018)
- [Personal VAD: Speaker-Conditioned Voice Activity Detection](https://mlsanity.com/paper/1908.04284/) (2019)
- [Overcoming Language Variation in Sentiment Analysis with Social Attention](https://mlsanity.com/paper/1511.06052/) (2015)
- [SkipFlow: Incorporating Neural Coherence Features for End-to-End Automatic Text Scoring](https://mlsanity.com/paper/1711.04981/) (2017)
- [Get It Scored Using AutoSAS -- An Automated System for Scoring Short Answers](https://mlsanity.com/paper/2012.11243/) (2020)
- [Controlling Personality-Based Stylistic Variation with Neural Natural Language Generators](https://mlsanity.com/paper/1805.08352/) (2018)
- [SkinAugment: Auto-Encoding Speaker Conversions for Automatic Speech Translation](https://mlsanity.com/paper/2002.12231/) (2020)
- [My Teacher Thinks The World Is Flat! Interpreting Automatic Essay Scoring Mechanism](https://mlsanity.com/paper/2012.13872/) (2020)
- [Multi-modal Automated Speech Scoring using Attention Fusion](https://mlsanity.com/paper/2005.08182/) (2020)
- [Towards Modelling Coherence in Spoken Discourse](https://mlsanity.com/paper/2101.00056/) (2020)

## Built on by (most cited first, 8 of 8)

- [Exploring the Capabilities of Prompted Large Language Models in Educational and Assessment Applications](https://mlsanity.com/paper/2405.11579/) (2024)
- [Automated Speech Scoring System Under The Lens: Evaluating and interpreting the linguistic cues for language proficiency](https://mlsanity.com/paper/2111.15156/) (2021)
- [AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses](https://mlsanity.com/paper/2109.11728/) (2021)
- [Using Sampling to Estimate and Improve Performance of Automated Scoring Systems with Guarantees](https://mlsanity.com/paper/2111.08906/) (2021)
- [Addressing Cold Start Problem for End-to-end Automatic Speech Scoring](https://mlsanity.com/paper/2306.14310/) (2023)
- [Towards Smarter Hiring: Are Zero-Shot and Few-Shot Pre-trained LLMs Ready for HR Spoken Interview Transcript Analysis?](https://mlsanity.com/paper/2504.05683/) (2025)
- [Automatic Proficiency Assessment in L2 English Learners](https://mlsanity.com/paper/2505.02615/) (2025)
- [MuFFIN: Multifaceted Pronunciation Feedback Model with Interactive Hierarchical Neural Modeling](https://mlsanity.com/paper/2510.04956/) (2025)

## Abstract (arXiv)

Automatic Speech Scoring (ASS) is the computer-assisted evaluation of a candidate's speaking proficiency in a language. ASS systems face many challenges like open grammar, variable pronunciations, and unstructured or semi-structured content. Recent deep learning approaches have shown some promise in this domain. However, most of these approaches focus on extracting features from a single audio, making them suffer from the lack of speaker-specific context required to model such a complex task. We propose a novel deep learning technique for non-native ASS, called speaker-conditioned hierarchical modeling. In our technique, we take advantage of the fact that oral proficiency tests rate multiple responses for a candidate. We extract context vectors from these responses and feed them as additional speaker-specific context to our network to score a particular response. We compare our technique with strong baselines and find that such modeling improves the model's average performance by 6.92% (maximum = 12.86%, minimum = 4.51%). We further show both quantitative and qualitative insights into the importance of this additional context in solving the problem of ASS.