# An Investigation into Value Misalignment in LLM-Generated Texts for Cultural Heritage

arXiv: https://arxiv.org/abs/2501.02039 · HTML: https://mlsanity.com/paper/2501.02039/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 70 references that are in the map.
- First posted on arXiv 3 Jan 2025.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models](https://mlsanity.com/paper/2402.05044/) (2024)
- [Human-like Summarization Evaluation with ChatGPT](https://mlsanity.com/paper/2304.02554/) (2023)
- [A Survey on Federated Unlearning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/2310.20448/) (2023)
- [How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions](https://mlsanity.com/paper/2406.14805/) (2024)
- [Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learning](https://mlsanity.com/paper/2408.16482/) (2024)
- [Halu-J: Critique-Based Hallucination Judge](https://mlsanity.com/paper/2407.12943/) (2024)
- [An Investigation into Misuse of Java Security APIs by Large Language Models](https://mlsanity.com/paper/2404.03823/) (2024)
- [How Well Do LLMs Identify Cultural Unity in Diversity?](https://mlsanity.com/paper/2408.05102/) (2024)
- [AI Safety Landscape for Large Language Models: Taxonomy, State-of-the-art, and Future Directions](https://mlsanity.com/paper/2408.12935/) (2024)
- [Benchmarking Cognitive Domains for LLMs: Insights from Taiwanese Hakka Culture](https://mlsanity.com/paper/2409.01556/) (2024)
- [Exploring Large Language Models on Cross-Cultural Values in Connection with Training Methodology](https://mlsanity.com/paper/2412.08846/) (2024)

## Built on by (most cited first, 8 of 8)

- [Time Travel: A Comprehensive Benchmark to Evaluate LMMs on Historical and Cultural Artifacts](https://mlsanity.com/paper/2502.14865/) (2025)
- [Do Large Language Models Truly Understand Cross-cultural Differences?](https://mlsanity.com/paper/2512.07075/) (2025)
- [Hanfu-Bench: A Multimodal Benchmark on Cross-Temporal Cultural Understanding and Transcreation](https://mlsanity.com/paper/2506.01565/) (2025)
- [CHisAgent: A Multi-Agent Framework for Event Taxonomy Construction in Ancient Chinese Cultural Systems](https://mlsanity.com/paper/2601.05520/) (2026)
- [Attribution Techniques for Mitigating Hallucinated Information in RAG Systems: A Survey](https://mlsanity.com/paper/2601.19927/) (2026)
- [Can Large Language Models Make Everyone Happy?](https://mlsanity.com/paper/2602.11091/) (2026)
- [Are Aligned Large Language Models Still Misaligned?](https://mlsanity.com/paper/2602.11305/) (2026)
- [Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branches](https://mlsanity.com/paper/2606.01145/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) become increasingly prevalent in tasks related to cultural heritage, such as generating descriptions of historical monuments, translating ancient texts, preserving oral traditions, and creating educational content, their ability to produce accurate and culturally aligned texts is being increasingly relied upon by users and researchers. However, cultural value misalignments may exist in generated texts, such as the misrepresentation of historical facts, the erosion of cultural identity, and the oversimplification of complex cultural narratives, which may lead to severe consequences. Therefore, investigating value misalignment in the context of LLM for cultural heritage is crucial for mitigating these risks, yet there has been a significant lack of systematic and comprehensive study and investigation in this area. To fill this gap, we systematically assess the reliability of LLMs in generating culturally aligned texts for cultural heritage-related tasks. We conduct a comprehensive evaluation by compiling an extensive set of 1066 query tasks covering 5 widely recognized categories with 17 aspects within the knowledge framework of cultural heritage across 5 open-source LLMs, and examine both the type and rate of cultural value misalignments in the generated texts. Using both automated and manual approaches, we effectively detect and analyze the cultural value misalignments in LLM-generated texts. Our findings are concerning: over 65% of the generated texts exhibit notable cultural misalignments, with certain tasks demonstrating almost complete misalignment with key cultural values. Beyond these findings, this paper introduces a benchmark dataset and a comprehensive evaluation workflow that can serve as a valuable resource for future research aimed at enhancing the cultural sensitivity and reliability of LLMs.