mlsanityData to 14 Oct 2026

Metric-Construction Coupling Inflates Measured Synthetic Dialect Recovery

arXiv 2610.00164 ↗First posted 14 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Korean dialect corpora are available but not redistributable: weights may be released, while reproducing training and evaluation from the underlying data cannot be. We ask how much of that supervision synthetic data recovers, and whether that recovery can be measured independently of the synthesis pipeline. We contribute KoDialectBench, 1,000 items across five regions on three axes, released as identifier hashes and scoring code so users reconstruct the items from their own licensed copy. Recovery is strongly axis-dependent: our best synthetic arm reaches 91.2% of the real-data gain on region identification but 63.7% on comprehension. On generation the answer depends on the metric: the deployed marker lexicon reports 92.3% on dialectness and 119.3% on region match, the latter exceeding the real-data reference, whereas reference-based generation reaches 72.4%. We find the marker metrics' scoring inventory is entirely contained in the inventory our transformation rules can emit. We test the effect of construction access directly with an exact-form construction-disjoint arm that withholds 20% of marker types from the rules. At exactly matched training size (8,600 examples) it reduces dialectness recovery from 91.8% to 8.1% and region-match recovery from 101.9% to 25.6% on the held-out marker inventory, while the three pipeline-independent measurements do not fall at all. A complementary evaluator sweep defines metric-construction coverage (MCC) and finds measured dialectness recovery increasing monotonically as overlap rises from MCC=0 to MCC=1. Shared construction and evaluation inventories can therefore substantially inflate estimates of synthetic-data recovery.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

39 of 51 references.

References · 39

  1. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  2. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  3. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  4. 2017Real-valued (Medical) Time Series Generation with Recurrent Conditional GANscited by 333
  5. 2019Measurement and Fairnesscited by 308
  6. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 295
  7. 2023Self-Consuming Generative Models Go MADcited by 250
  8. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  9. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 196
  10. 2018Categorizing Variants of Goodhart's Lawcited by 149
  11. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
  12. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 100
27 more
  1. 2019Evaluating Style Transfer for Textcited by 61
  2. 2024Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discriminationcited by 56
  3. 2020Investigating African-American Vernacular English in Transformer-Based Text Generationcited by 53
  4. 2022Multi-VALUE: A Framework for Cross-Dialectal English NLPcited by 53
  5. 2023An Open Source Data Contamination Report for Large Language Modelscited by 51
  6. 2022VALUE: Understanding Dialect Disparity in NLUcited by 47
  7. 2024Evaluating Language Models as Synthetic Data Generatorscited by 33
  8. 2021Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfercited by 25
  9. 2024Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Taskscited by 22
  10. 2019Style Transfer for Texts: Retrain, Report Errors, Compare with Rewritescited by 20
  11. 2024Data Contamination Report from the 2024 CONDA Shared Taskcited by 20
  12. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 20
  13. 2024Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluationcited by 19
  14. 2024Quality Matters: Evaluating Synthetic Data for Tool-Using LLMscited by 19
  15. 2023A Call for Standardization and Validation of Text Style Transfer Evaluationcited by 14
  16. 2023DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rulescited by 13
  17. 2023Text Style Transfer Evaluation Using Large Language Modelscited by 13
  18. 2022Human Judgement as a Compass to Navigate Automatic Metrics for Formality Transfercited by 12
  19. 2024Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varietiescited by 12
  20. 2023Goodhart's Law Applies to NLP's Explanation Benchmarkscited by 10
  21. 2025What Has Been Lost with Synthetic Evaluation?cited by 9
  22. 2025Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?cited by 7
  23. 2025EnDive: A Cross-Dialect Benchmark for Fairness and Performance in Large Language Modelscited by 5
  24. 2025Fine-Tuning LLMs for Low-Resource Dialect Translation: The Case of Lebanesecited by 3
  25. 2025A Rigorous Evaluation of LLM Data Generation Strategies for Low-Resource Languagescited by 3
  26. 2025Mind the Style Gap: Meta-Evaluation of Style and Attribute Transfer Metricscited by 2
  27. 2025Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translationcited by 2

Built on by

Not cited yet.