mlsanityData to 7 Oct 2026

CSPF: A Constrained Shared-Private Fusion Method for Non-Verifiable Preference Evaluation

arXiv 2607.20862 ↗First posted 23 Jul 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

At present, reliable evaluation of non-verifiable tasks remains challenging. Existing approaches often fail to adequately capture the diverse evaluative criteria underlying human preferences in such tasks. To this end, we propose Constrained Shared-Private Fusion (CSPF), a fusion method that treats heterogeneous frozen reward models as complementary evaluators and learns to integrate their hidden-state representations under pairwise human-preference supervision. CSPF decomposes each expert signal into shared and expert-private representations, encouraging cross-expert alignment while preserving complementary viewpoints. Across experiments on LM-Arena target-domain adaptation and PPE out-of-distribution preference evaluation, CSPF achieves the best performance on the primary metrics among the evaluated single-expert reward-model, scalar-score multi-expert, and rubric-judge baselines. Overall, CSPF suggests that fusing hidden-state representations provides a more expressive basis for preference assessment, offering a practical route toward integrated evaluative signals for non-verifiable preference tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 23 Jul 2026.

Builds on

23 of 32 references.

References · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2020Supervised Contrastive Learningcited by 2,803
  4. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  5. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  6. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  7. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  8. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  9. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  10. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  11. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  12. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
11 more
  1. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  2. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  3. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  4. 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesiscited by 89
  5. 2024How to Evaluate Reward Models for RLHFcited by 84
  6. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  7. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 55
  8. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 47
  9. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22
  10. 2021groupShapley: Efficient prediction explanation with Shapley values for feature groupscited by 21
  11. 2025Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Saycited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modeling
  2. 2026VRM: Teaching Reward Models to Understand Authentic Human Preferences
  3. 2026In-Context Reward Adaptation for Robust Preference Modeling
  4. 2026Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learning
  5. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking
  6. 2026Variance-aware Reward Modeling with Anchor Guidance
  7. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models
  8. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences