mlsanityData to 7 Oct 2026

Marginal Alignment Does Not Guarantee Joint-Distribution Fidelity: An Official-Reference Audit of Nemotron-Personas-Korea with Cross-Locale Replication

arXiv 2606.12433 ↗First posted 15 May 2026cs.CY cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Synthetic persona datasets cite alignment with official demographics as a basis for trust, yet downstream users consume them as joint structures across age, sex, region, occupation, education, name, and institutional status. Marginal alignment does not imply that these joints are preserved. We propose the Independence-Assumption Footprint (IAF), an audit primitive that operates on the attribute combinations a dataset card itself documents as treated independently. For each such combination, IAF compares the synthetic joint against an external official or institutional reference, using direct joint tables where available and rule-implied checks otherwise. Applied to NVIDIA Nemotron-Personas-Korea (one million Korean synthetic personas), IAF finds that NPK aligns with KOSIS marginals while three joints fail. The major-by-occupation distribution against the KEIS graduate universe carries a large conditional mismatch. The age profile of military service is institutionally inconsistent. Female representation in male-dominated occupations is substantially over-flattened toward parity, with the strict screening verdict mapping-dependent and age-robust under direct standardisation. A transferability demonstration across six further NPK locales finds locale-dependent rather than universal diagnostics, with reference-taxonomy cardinality confounding cross-locale flag counts. For synthetic personas used as silicon samples, marginal claims must therefore be paired with disclosure-anchored joint audits before reuse. The released audit artefacts (reference manifests, occupational crosswalks, derived metrics, reproducibility scripts) instantiate this protocol on the NPK family and are released for retargeting at other synthetic persona resources.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 81 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

43 of 81 references.

References · 43

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  3. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  4. 2018Datasheets for Datasetscited by 1,371
  5. 2018Model Cards for Model Reportingcited by 1,068
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  7. 2023Whose Opinions Do Language Models Reflect?cited by 759
  8. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  9. 2014Certifying and removing disparate impactcited by 715
  10. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  11. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 671
  12. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
31 more
  1. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  2. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  3. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  4. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  5. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  6. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  7. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  8. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  9. 2019A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cyclecited by 192
  10. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 169
  11. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  12. 2020Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructurecited by 138
  13. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
  14. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  15. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 94
  16. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  17. 2025LLM Generated Persona is a Promise with a Catchcited by 77
  18. 2023KoBBQ: Korean Bias Benchmark for Question Answeringcited by 71
  19. 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Modelscited by 47
  20. 2023KoSBi: A Dataset for Mitigating Social Bias Risks Towards Safer Large Language Model Applicationcited by 24
  21. 2025Population-Aligned Persona Generation for LLM-based Social Simulationcited by 20
  22. 2021Predicting Lexical Complexity in English Texts: The Complex 2.0 Datasetcited by 15
  23. 2025Biased Tales: Cultural and Topic Bias in Generating Children's Storiescited by 15
  24. 2022KOLD: Korean Offensive Language Datasetcited by 14
  25. 2025Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparencycited by 14
  26. 2023Auditing and Generating Synthetic Data with Controllable Trust Trade-offscited by 13
  27. 2025Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactionscited by 12
  28. 2024Subtle Biases Need Subtler Measures: Dual Metrics for Evaluating Representative and Affinity Bias in Large Language Modelscited by 10
  29. 2025ConsistencyAI: A Benchmark to Assess LLMs' Factual Consistency When Responding to Different Demographic Groupscited by 2
  30. 2026When LLMs Imagine People: A Human-Centered Persona Brainstorm Audit for Bias and Fairness in Creative Applicationscited by 2
  31. 2026The Personality Trap: How LLMs Embed Bias When Generating Human-Like Personascited by 1

Built on by

Not cited yet.