# Beyond Value Benchmarks: Measuring Value-Structure Alignment in Large Language Models via Symmetric Q-Sorts

arXiv: https://arxiv.org/abs/2606.21939 · HTML: https://mlsanity.com/paper/2606.21939/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 77 references that are in the map.
- First posted on arXiv 20 Jun 2026.

## Builds on (most cited first, 15 of 15)

- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Social Chemistry 101: Learning to Reason about Social and Moral Norms](https://mlsanity.com/paper/2011.00620/) (2020)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals](https://mlsanity.com/paper/2210.01790/) (2022)
- [Moral Foundations of Large Language Models](https://mlsanity.com/paper/2310.15337/) (2023)
- [Benchmarking Distributional Alignment of Large Language Models](https://mlsanity.com/paper/2411.05403/) (2024)
- [Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?](https://mlsanity.com/paper/2501.15463/) (2025)
- [MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Tasks](https://mlsanity.com/paper/2310.19677/) (2023)
- [Exploring Multilingual Concepts of Human Value in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?](https://mlsanity.com/paper/2402.18120/) (2024)
- [Structural Similarities Between Language Models and Neural Response Measurements](https://mlsanity.com/paper/2306.01930/) (2023)

## Built on by (most cited first, 1 of 1)

- [GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models](https://mlsanity.com/paper/2609.18384/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly deployed in contexts requiring complex moral reasoning and value trade-offs. However, existing evaluations typically rely on item-level behavioral metrics, which fail to capture how models structurally prioritize competing values as a cohesive system. To address this, we propose a symmetric human-LLM evaluation framework, grounded in Q methodology, to measure value-structure alignment. Under our protocol, humans and models sort an identical 140-item moral statement set into a shared nine-column forced distribution; for LLMs, we elicit strict rankings and deterministically map them to Q-sort buckets. Using a human reference sample (N=35), we establish a stable three-factor reference geometry specific to this instrument and sample. We evaluate 12 LLMs across four model families via 240 replicated Q-sorts at two temperature settings, quantifying structural alignment via Procrustes similarity (φ) and RSA-based Spearman correlation (ρ). Our results reveal significant cross-family heterogeneity, model-specific sensitivity to generation stochasticity and localized misalignment, which demonstrate that favorable global scores can obscure underlying regional distortions. While rank- and bucket-based analyses remain highly consistent, prompt phrasing introduces notable variance. Ultimately, assessing value-structure alignment provides a crucial structural complement to traditional itemwise moral benchmarks.