mlsanityData to 7 Oct 2026

DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4

arXiv 2305.14702 ↗First posted 24 May 2023cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Human preference judgments are pivotal in guiding large language models (LLMs) to produce outputs that align with human values. Human evaluations are also used in summarization tasks to compare outputs from various systems, complementing existing automatic metrics. Despite their significance, however, there has been limited research probing these pairwise or k-wise comparisons. The collective impact and relative importance of factors such as output length, informativeness, fluency, and factual consistency are still not well understood. It is also unclear if there are other hidden factors influencing human judgments. In this paper, we conduct an in-depth examination of a collection of pairwise human judgments released by OpenAI. Utilizing the Bradley-Terry-Luce (BTL) model, we reveal the inherent preferences embedded in these human judgments. We find that the most favored factors vary across tasks and genres, whereas the least favored factors tend to be consistent, e.g., outputs are too brief, contain excessive off-focus content or hallucinated facts. Our findings have implications on the construction of balanced datasets in human preference evaluations, which is a crucial step in shaping the behaviors of future LLMs.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 38 of its 69 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

38 of 69 references.

References · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  8. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  9. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  10. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  11. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  12. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
26 more
  1. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  2. 2022Teaching language models to support answers with verified quotescited by 290
  3. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  4. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  5. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  6. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  7. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  8. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  9. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  10. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 128
  11. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  12. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  13. 2018Multi-Reward Reinforced Summarization with Saliency and Entailmentcited by 73
  14. 2019Generating Personalized Recipes from Historical User Preferencescited by 52
  15. 2021Better than Average: Paired Evaluation of NLP Systemscited by 30
  16. 2022Multimodal Knowledge Alignment with Reinforcement Learningcited by 30
  17. 2021The Summary Loop: Learning to Write Abstractive Summaries Without Examplescited by 27
  18. 2023Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluationcited by 24
  19. 2021Reinforcement Learning for Abstractive Question Summarization with Question-aware Semantic Rewardscited by 21
  20. 2017A practical guide and software for analysing pairwise comparison experimentscited by 18
  21. 2021Refocusing on Relevance: Personalization in NLGcited by 18
  22. 2020Understanding Points of Correspondence between Sentences for Abstractive Summarizationcited by 16
  23. 2023Prefix Propagation: Parameter-Efficient Tuning for Long Sequencescited by 15
  24. 2020Learning to Fuse Sentences with Transformers for Summarizationcited by 14
  25. 2019Guiding Extractive Summarization with Question-Answering Rewardscited by 13
  26. 2022Parameter-Efficient Legal Domain Adaptationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 4; 2026: 1.

16 papers.

Sort:

By area:

  1. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  2. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  3. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  4. 2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignmentcited by 24
  5. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  6. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  7. 2024SportsMetrics: Blending Text and Numerical Data to Understand Information Fusion in LLMscited by 8
  8. 2024AFSPP: Agent Framework for Shaping Preference and Personality with Large Language Modelscited by 7
  9. 2025AdParaphrase: Paraphrase Dataset for Analyzing Linguistic Features toward Generating Attractive Ad Textscited by 5
  10. 2023Rescue: Ranking LLM Responses with Partial Ordering to Improve Response Generationcited by 4
  11. 2024DeFine: Decision-Making with Analogical Reasoning over Factor Profilescited by 3
  12. 2025Rethinking Human Preference Evaluation of LLM Rationalescited by 3
4 more
  1. 2024Uncovering Factor Level Preferences to Improve Human-Model Alignmentcited by 2
  2. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  3. 2025From Fragments to Facts: A Curriculum-Driven DPO Approach for Generating Hindi News Veracity Explanationscited by 1
  4. 2025Improving the End-to-End Efficiency of Offline Inference for Multi-LLM Applications Based on Sampling and Simulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
  2. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators
  3. 2024A Survey on Human Preference Learning for Large Language Models
  4. 2023Can LLMs Capture Human Preferences?
  5. 2026How Many Human Judgments Are Enough? Feasibility Limits of Human Preference Evaluation
  6. 2024Dissecting Human and LLM Preferences
  7. 2025Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?
  8. 2025WorldPM: Scaling Human Preference Modeling