mlsanityData to 14 Oct 2026

Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on Consistency with Human Preferences

arXiv 2303.07610 ↗First posted 14 Mar 2023cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

As a natural language assistant, ChatGPT is capable of performing various tasks, including but not limited to article generation, code completion, and data analysis. Furthermore, ChatGPT has consistently demonstrated a remarkable level of accuracy and reliability in terms of content evaluation, exhibiting the capability of mimicking human preferences. To further explore ChatGPT's potential in this regard, a study is conducted to assess its ability to rank content. In order to do so, a test set consisting of prompts is created, covering a wide range of use cases, and five models are utilized to generate corresponding responses. ChatGPT is then instructed to rank the responses generated by these models. The results on the test set show that ChatGPT's ranking preferences are consistent with human to a certain extent. This preliminary experimental finding implies that ChatGPT's zero-shot ranking capability could be used to reduce annotation pressure in a number of ranking tasks.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 14 Mar 2023.

Builds on

39 of 46 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
27 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  12. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  13. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  14. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  15. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  16. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  17. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  18. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  19. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  20. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  21. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  22. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  23. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  24. 2023Pretraining Language Models with Human Preferencescited by 247
  25. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 154
  26. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 130
  27. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 36

Built on by

Cited by year

Citing papers per year: 2023: 16; 2024: 8; 2025: 1.

25 papers.

Sort:

By area:

  1. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 274
  2. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  3. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  4. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 94
  5. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 93
  6. 2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 86
  7. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  8. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  9. 2023CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translationcited by 57
  10. 2023"HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social mediacited by 45
  11. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  12. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructionscited by 34
13 more
  1. 2023A Preliminary Study of the Intrinsic Relationship between Complexity and Alignmentcited by 33
  2. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  3. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  4. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 18
  5. 2023ChatGPT is a Remarkable Tool -- For Expertscited by 16
  6. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  7. 2023ChatGPT Hallucinates when Attributing Answerscited by 14
  8. 2023Generating Natural Language Queries for More Effective Systematic Review Screening Prioritisationcited by 6
  9. 2024Explingo: Explaining AI Predictions using Large Language Modelscited by 6
  10. 2024Language Models can Evaluate Themselves via Probability Discrepancycited by 5
  11. 2024MD tree: a model-diagnostic tree grown on loss landscapecited by 3
  12. 2024Large Language Models Meet Graph Neural Networks: A Perspective of Graph Miningcited by 3
  13. 2025Implementing Rational Choice Functions with LLMs and Measuring their Alignment with User Preferencescited by 0