mlsanityData to 7 Oct 2026

LocalValueBench: A Collaboratively Built and Extensible Benchmark for Evaluating Localized Value Alignment and Ethical Safety in Large Language Models

arXiv 2408.01460 ↗First posted 27 Jul 2024cs.CY cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

The proliferation of large language models (LLMs) requires robust evaluation of their alignment with local values and ethical standards, especially as existing benchmarks often reflect the cultural, legal, and ideological values of their creators. textscLocalValueBench, introduced in this paper, is an extensible benchmark designed to assess LLMs' adherence to Australian values, and provides a framework for regulators worldwide to develop their own LLM benchmarks for local value alignment. Employing a novel typology for ethical reasoning and an interrogation approach, we curated comprehensive questions and utilized prompt engineering strategies to probe LLMs' value alignment. Our evaluation criteria quantified deviations from local values, ensuring a rigorous assessment process. Comparative analysis of three commercial LLMs by USA vendors revealed significant insights into their effectiveness and limitations, demonstrating the critical importance of value alignment. This study offers valuable tools and methodologies for regulators to create tailored benchmarks, highlighting avenues for future research to enhance ethical AI development.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 26 of its 34 references that are in the map. First posted on arXiv 27 Jul 2024.

Builds on

26 of 34 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  5. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  6. 2023Scaling Data-Constrained Language Modelscited by 391
  7. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  8. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  9. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  10. 2023Improving CLIP Training with Language Rewritescited by 216
  11. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
  12. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 188
14 more
  1. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 179
  2. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  3. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 143
  4. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  5. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
  6. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
  7. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  8. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 65
  9. 2022The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMscited by 57
  10. 2022BARS: Towards Open Benchmarking for Recommender Systemscited by 54
  11. 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
  12. 2022BigBIO: A Framework for Data-Centric Biomedical Natural Language Processingcited by 36
  13. 2020Algorithmic Fairness from a Non-ideal Perspectivecited by 30
  14. 2023LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarioscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 1.

7 papers.

  1. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  2. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  3. 2025Generative Psycho-Lexical Approach for Constructing Value Systems in Large Language Modelscited by 4
  4. 2025AdAEM: An Adaptively and Automated Extensible Measurement of LLMs' Value Differencecited by 4
  5. 2025Supernova Event Dataset: Interpreting Large Language Models' Personality through Critical Event Analysiscited by 1
  6. 2025CALMA: A Process for Deriving Context-aligned Axes for Language Model Alignmentcited by 0
  7. 2026Toward Agentic Optical Networks: A Vision of LLM Agent-Driven Autonomous Lifecycle Managementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Diverse Human Value Alignment for Large Language Models via Ethical Reasoning
  2. 2026ContextAdapt: Evaluating Contextual Adaptation and Value Alignment in LLMs
  3. 2024MoralBench: Moral Evaluation of LLMs
  4. 2025Benchmarking Multi-National Value Alignment for Large Language Models
  5. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Survey
  6. 2026Beyond Value Benchmarks: Measuring Value-Structure Alignment in Large Language Models via Symmetric Q-Sorts
  7. 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility
  8. 2023Flames: Benchmarking Value Alignment of LLMs in Chinese