Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment
On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment
Abstract (arXiv)
Human-feedback alignment has made language models useful assistants and is commonly described as aligning them with humans. However, the responses people prefer from an AI need not be the responses they themselves would give. We distinguish alignment with human preferences from alignment with human behavior, and show that alignment with human preferences can make model behavior less human-like even when both preferences and responses come entirely from humans. We call this the Turing-test gap. We show that preference alignment preserves the human response distribution only under a restrictive condition, and find no consistent evidence that real human preferences satisfy it. Empirically, the loss of human-response likelihood increases with the strength of preference weighting, regardless of its direction, and the gap also appears under standard DPO. These results establish human-likeness as an explicit dimension of alignment rather than something assumed to follow from preference alignment.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 64 references that are in the map. First posted on arXiv 20 Sep 2026.
Builds on
43 of 64 references.
Closest ancestors · 2
Same direction, newest first
- 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
- 2020Learning to summarize from human feedback2 Sep
Other references · 41
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
- 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
- 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
- 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
29 more
- 2023Whose Opinions Do Language Models Reflect?cited by 773
- 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
- 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
- 2020Recipes for building an open-domain chatbotcited by 593
- 2019ELI5: Long Form Question Answeringcited by 575
- 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
- 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
- 2020Towards a Human-like Open-Domain Chatbotcited by 470
- 2018Scalable agent alignment via reward modeling: a research directioncited by 447
- 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
- 2016Cooperative Inverse Reinforcement Learningcited by 300
- 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
- 2021TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generationcited by 120
- 2023Ghostbuster: Detecting Text Ghostwritten by Large Language Modelscited by 91
- 2024Delving into LLM-assisted writing in biomedical publications through excess vocabularycited by 78
- 2024Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Editscited by 49
- 2025Large Language Models Pass the Turing Testcited by 48
- 2025People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated textcited by 40
- 2024How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?cited by 27
- 2023Human or Not? A Gamified Approach to the Turing Testcited by 16
- 2025HumT DumT: Measuring and controlling human-like language in LLMscited by 12
- 2026Understanding Diversity Collapse in RLVR via the Lens of Overtrainingcited by 10
- 2026MatrAIx: Simulating the World with 8.3 Billion Persona Agentscited by 7
- 2024X-TURING: Towards an Enhanced and Efficient Turing Test for Long-Term Dialogue Agentscited by 6
- 2025Computational Turing Test Reveals Systematic Differences Between Human and AI Languagecited by 6
- 2026Post-training makes large language models less human-likecited by 5
- 2025Is Human-Like Text Liked by Humans? Multilingual Human Detection and Preference Against AIcited by 4
- 2026HAL: Inducing Human-likeness in LLMs with Alignmentcited by 1
- 2026Mitigating Mismatch within Reference-based Preference Optimizationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Understanding the Learning Dynamics of Alignment with Human Feedback
- 2025HPS: Hard Preference Sampling for Human Preference Alignment
- 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback
- 2026AI Alignment From Social Choice Perspectives
- 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
- 2025Learning a Canonical Basis of Human Preferences from Binary Ratings
- 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
- 2026Alignment Makes Language Models Normative, Not Descriptive