mlsanityData to 14 Oct 2026

Prior over Evidence: Stereotype-Driven Diagnosis in LLM-Based L2 Pronunciation Feedback

arXiv 2606.15325 ↗First posted 13 Jun 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models are increasingly deployed for written pronunciation feedback in second-language (L2) English learning, under the assumption that their diagnoses are grounded in the supplied speech evidence rather than in priors from pretraining. This assumption is tested on 1,800 L2-Arctic utterances spanning six L1 backgrounds, three audio-capable LLMs, four pronunciation dimensions, and five evidence conditions ranging from a text-only baseline to numeric acoustic features and raw audio. Each (utterance x model x condition x dimension) cell is scored on three metrics: Rating Accuracy (RA) against gold labels, Evidence Coherence (EC) assessing internal consistency without ground truth, and Grounded Correctness (GC) evaluated against gold evidence. Results show three findings across models. First, rating accuracy and grounded reasoning decouple: 39.6% of judged cells contain internally coherent reasoning that supports a wrong rating, against only 15.8% where the reasoning supports a correct rating. Second, phoneme-level feedback converges to a fixed inventory of L2-English difficulty phones that recurs across all six L1 backgrounds and all evidence conditions. Third, acoustic evidence improves the rating only when the supplied feature directly probes the target dimension: textualised F0 range raises pitch-variation grounding from (0.18-0.19) to (0.45-0.62) across all three models, while stress and phoneme correctness, which require target-to-realisation alignment, remain ungrounded. The same audio waveform without textualised F0 values does not reproduce this improvement. These findings indicate that current general-purpose LLMs are more reliable as verbalisers of externally computed pronunciation evidence than as standalone diagnostic engines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 28 references that are in the map. First posted on arXiv 13 Jun 2026.

Builds on

13 of 28 references.

References · 13

  1. 2019Language Models as Knowledge Bases?cited by 1,936
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  3. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  4. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  5. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  6. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  7. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
  8. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 243
  9. 2023Faithfulness Tests for Natural Language Explanationscited by 107
  10. 2025Assessment of L2 Oral Proficiency using Speech Large Language Modelscited by 14
  11. 2025Read to Hear: A Zero-Shot Pronunciation Assessment Using Textual Descriptions and LLMscited by 5
  12. 2025Exploring the Potential of Large Multimodal Models as Effective Alternatives for Pronunciation Assessmentcited by 4
1 more
  1. 2025ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Modelscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedback
  2. 2026Hearing Between the Lines: Unlocking the Reasoning Power of LLMs for Speech Evaluation
  3. 2026Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accents
  4. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning
  5. 2025Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation
  6. 2025AHELM: A Holistic Evaluation of Audio-Language Models
  7. 2026Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modeling
  8. 2025How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidence