Toward Cultural Alignment: Human-Centered Evaluation of Multimodal AI Stories Across Five African Communities
Not on the map. Why?
Abstract (arXiv)
In this paper, we examine how well AI-generated multimodal stories align with the lived practices, relationships, language, values, and visual expectations of the communities they represent. We conduct a community-grounded mixed-methods evaluation with 19 culture representatives across five African communities, combining quantitative annotations with qualitative focus group discussions. We find that cultural alignment depends not simply on recognizable cultural markers, but on how those markers fit social, linguistic, procedural, and visual context. From these evaluations, we develop a taxonomy of cultural alignment comprising five broader cultural marker categories and eight recurring mechanisms of misalignment. We additionally evaluate five multimodal LLM judges to examine whether automated evaluation can approximate community-grounded judgments at scale. Judge reliability and score calibration vary substantially across communities, with no single judge performing consistently across all five settings. These findings motivate community-calibrated evaluation pipelines in which automated judges are validated against community judgments to determine where they can be trusted and where human review remains necessary.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 41 references that are in the map. First posted on arXiv 29 Aug 2026.
Builds on
22 of 41 references.
References · 22
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
- 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
- 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
- 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
- 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 206
- 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 167
- 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 149
- 2024Are Large Language Models Capable of Generating Human-Level Narratives?cited by 79
- 2024AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuancescited by 60
- 2025Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMscited by 54
- 2024NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Modelscited by 48
10 more
- 2023NormDial: A Comparable Bilingual Synthetic Dialog Dataset for Modeling Social Norm Adherence and Violationcited by 24
- 2025CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metricscited by 15
- 2024RENOVI: A Benchmark Towards Remediating Norm Violations in Socio-Cultural Conversationscited by 14
- 2024Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?cited by 10
- 2025Fluent but Foreign: Even Regional LLMs Lack Cultural Alignmentcited by 10
- 2025Culture in Action: Evaluating Text-to-Image Models through Social Activitiescited by 10
- 2025TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Storiescited by 10
- 2025CuRe: Cultural Gaps in the Long Tail of Text-to-Image Systemscited by 8
- 2024Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representationcited by 7
- 2026Evaluating AI-Generated Images of Cultural Artifacts with Community-Informed Rubricscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025'Too much alignment; not enough culture': Re-balancing cultural alignment practices in LLMs
- 2025Toward Socially Aware Vision-Language Models: Evaluating Cultural Competence Through Multimodal Story Generation
- 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
- 2024CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge
- 2025Cultural Bias in Large Language Models: Evaluating AI Agents through Moral Questionnaires
- 2025Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Models
- 2023Cultural Bias and Cultural Alignment of Large Language Models
- 2024Survey of Cultural Awareness in Language Models: Text and Beyond