mlsanityData to 14 Oct 2026

DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American English

arXiv 2601.22888 ↗First posted 30 Jan 2026cs.CL cs.AIcited by 0

On the map: Mechanistic interpretability → Knowledge and representations inside LLMs

Abstract (arXiv)

More than 80% of the 1.6B English speakers do not use Standard American English (SAE), yet LLMs often fail to correctly identify non-SAE dialects and generate stereotyped responses for their speakers. We introduce DialectLLM, the first large-scale framework for generating high-quality multi-dialectal conversational data encompassing the three pillars of written dialect -- lexical (vocabulary), orthographic (spelling), and morphosyntactic (grammar) features. DialectLLM produces a dialect-parallel dialog dataset spanning nine English dialects. Partnering with native linguists, we design and validate SAE-to-dialect transformation rules, ensuring authenticity. Our approach challenges the prevailing practice of applying a single morphosyntactic feature set to both user utterances and model responses, showing that models should not reproduce up to 90% of the grammatical features of a dialect. Human evaluation confirms data quality, with annotators preferring DialectLLM over prior methods in 98.8% of pairwise comparisons for dialect naturalness. We then construct DialectLLM-Bench, a dialect-parallel benchmark with 50k+ dialogs, resulting in 97k+ QA pairs, and evaluate 17 LLMs on dialect identification and response generation tasks. Even frontier models achieve under 70% accuracy, fail to reach 50% for prominent dialects like Canadian English, and systematically misclassify non-SAE dialects as American or British. Beyond benchmarking, we show that DialectLLM data also serve as a scalable LLM post-training resource, suggesting a practical path toward dialect-aware conversational AI.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability. Builds on 25 of its 33 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

25 of 33 references.

References · 25

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2025Gemma 3 Technical Reportcited by 1,824
  4. 2024A Survey on LLM-as-a-Judgecited by 1,423
  5. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  6. 2024POSIX: A Prompt Sensitivity Index For Large Language Modelscited by 59
  7. 2024Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discriminationcited by 56
  8. 2022Multi-VALUE: A Framework for Cross-Dialectal English NLPcited by 53
  9. 2023Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languagescited by 51
  10. 2024DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languagescited by 50
  11. 2022VALUE: Understanding Dialect Disparity in NLUcited by 47
  12. 2024Natural Language Processing for Dialects of a Language: A Surveycited by 40
13 more
  1. 2023Evaluation of African American Language Bias in Natural Language Generationcited by 36
  2. 2024Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Taskscited by 22
  3. 2023TADA: Task-Agnostic Dialect Adapters for Englishcited by 15
  4. 2024Evaluating Dialect Robustness of Language Models via Conversation Understandingcited by 14
  5. 2023DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rulescited by 13
  6. 2023Task-Agnostic Low-Rank Adapters for Unseen English Dialectscited by 9
  7. 2024ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Modelscited by 9
  8. 2024Standard Language Ideology in AI-Generated Languagecited by 5
  9. 2025EnDive: A Cross-Dialect Benchmark for Fairness and Performance in Large Language Modelscited by 5
  10. 2025Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varietiescited by 4
  11. 2025Data Caricatures: On the Representation of African American Language in Pretraining Corporacited by 3
  12. 2025DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generationcited by 3
  13. 2025Finding A Voice: Exploring the Potential of African American Dialect and Voice Generation for Chatbotscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation
  2. 2026The Dialect Tax: Dialectal Biases Persist throughout the Language Modeling Pipeline
  3. 2025DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models
  4. 2025Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks
  5. 2026Can LLM Agents Identify Spoken Dialects like a Linguist?
  6. 2025Can MLLMs Generalize to Multi-Party dialog? Exploring Multilingual Response Generation in Complex Scenarios
  7. 2026A Dialectic Pipeline for Improving LLM Robustness
  8. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation