mlsanityData to 7 Oct 2026

Does Capability Transfer to Subjective Behavior -- and Would Our Instruments Tell Us? A Self-Evolving, Trust-by-Construction Evaluation Paradigm

arXiv 2605.27914 ↗First posted 27 May 2026cs.CL cs.AIcited by 0

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Benchmarking is mature where answers are verifiable -- math, code, reasoning -- but the fastest-growing uses of LLMs are subjective and human-facing: companionship, emotional support, counseling. There the default validity test, correlating a metric to human judgment, has no stable anchor: inter-rater agreement is low, structured by annotator identity, barely reproducible, and length-biased. So we cannot answer the question that matters: does capability that scales on objective benchmarks transfer to subjective behavior, and would our instruments even tell us if it did not? We build an instrument for this regime and report what it reveals at the frontier. We contribute, first, a self-evolving instrument that selects and then authors its own behavioral dimensions under a multiplicative anti-gaming fitness, self-halting when it stops improving; second, a trust-by-construction paradigm that earns belief through three certificates established without a human gold standard, where human raters saturate (rho ~ 0.45); and third, the finding it makes visible -- capability transfer is dissociable. Across 49 models, 8 families, and 24 months, subjective behaviors are where objective-benchmark scaling fails to carry over: the sharpest case, advice-restraint (knowing when not to give advice), is the frontier's universal-lowest dimension, and at gpt-4.1->gpt-5 it ran backwards while the aggregate score hid it -- a regression one instruction recovers. Warm restraint is moved by model generation, not by raw scale, MoE width, inference budget, or reasoning mode; the open-weight Pareto frontier matches closed flagships at ~10-80x lower per-call cost; and four judge families replicate the rubric on held-out human ESConv conversations. Data, code, the locked rubric, and judge prompts will be released upon publication.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 65 of its 147 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

65 of 147 references.

References · 65

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2024A Survey on LLM-as-a-Judgecited by 1,384
  10. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  11. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
38 more
  1. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  2. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  3. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  9. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  10. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  11. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  12. 2020Recipes for building an open-domain chatbotcited by 593
  13. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  14. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  15. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  16. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  17. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  18. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  19. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  20. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  21. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  22. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  23. 2022Defining and Characterizing Reward Hackingcited by 184
  24. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
  25. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
  26. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  27. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  28. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  29. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
  30. 2018Categorizing Variants of Goodhart's Lawcited by 132
  31. 2022Broken Neural Scaling Lawscited by 102
  32. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  33. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  34. 2023DICES Dataset: Diversity in Conversational AI Evaluation for Safetycited by 86
  35. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  36. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  37. 2024MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Modelscited by 78
  38. 2021Towards Facilitating Empathic Conversations in Online Mental Health Support: A Reinforcement Learning Approachcited by 75

50 most cited of 65 All 65 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluating Alignment of Behavioral Dispositions in LLMs
  2. 2025Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications
  3. 2024Trusting Your AI Agent Emotionally and Cognitively: Development and Validation of a Semantic Differential Scale for AI Trust
  4. 2026Selective Deficits in LLM Mental Self-Modeling in a Behavior-Based Test of Theory of Mind
  5. 2026Recognition, Simulation, and Refusal: A Contamination-Aware Study of Classic Psychological Effects in LLM Agents
  6. 2025A closer look at how large language models trust humans: patterns and biases
  7. 2025The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs
  8. 2026Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs