mlsanityData to 14 Oct 2026

Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4 Text Ratings

arXiv 2308.02575 ↗First posted 3 Aug 2023cs.CL cs.AIcited by 34

Not on the map. Why?

Abstract (arXiv)

This study investigates the consistency of feedback ratings generated by OpenAI's GPT-4, a state-of-the-art artificial intelligence language model, across multiple iterations, time spans and stylistic variations. The model rated responses to tasks within the Higher Education (HE) subject domain of macroeconomics in terms of their content and style. Statistical analysis was conducted in order to learn more about the interrater reliability, consistency of the ratings across iterations and the correlation between ratings in terms of content and style. The results revealed a high interrater reliability with ICC scores ranging between 0.94 and 0.99 for different timespans, suggesting that GPT-4 is capable of generating consistent ratings across repetitions with a clear prompt. Style and content ratings show a high correlation of 0.87. When applying a non-adequate style the average content ratings remained constant, while style ratings decreased, which indicates that the large language model (LLM) effectively distinguishes between these two criteria during evaluation. The prompt used in this study is furthermore presented and explained. Further research is necessary to assess the robustness and reliability of AI models in various use cases.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 14 of its 34 references that are in the map. First posted on arXiv 3 Aug 2023.

Builds on

14 of 34 references.

References · 14

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Emergent Abilities of Large Language Modelscited by 2,512
  3. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  4. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  5. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  6. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  7. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  8. 2023GPTScore: Evaluate as You Desirecited by 361
  9. 2023How is ChatGPT's behavior changing over time?cited by 270
  10. 2022Prompting GPT-3 To Be Reliablecited by 260
  11. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 224
  12. 2023Boosting Theory-of-Mind Performance in Large Language Models via Promptingcited by 64
2 more
  1. 2023TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Taskscited by 49
  2. 2022Learning to Perform Complex Tasks through Compositional Fine-Tuning of Language Modelscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 18; 2025: 7; 2026: 4.

34 papers.

Sort:

By area:

  1. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  2. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  3. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 159
  4. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 106
  5. 2023Mark My Words: Analyzing and Evaluating Language Model Watermarkscited by 71
  6. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43
  7. 2024HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematicscited by 32
  8. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 28
  9. 2024Reinforcement Learning for Optimizing RAG for Domain Chatbotscited by 26
  10. 2024Can AI grade your essays? A comparative analysis of large language models and teacher ratings in multidimensional essay scoringcited by 20
  11. 2023WeaverBird: Empowering Financial Decision-Making with Large Language Model, Knowledge Base, and Search Enginecited by 17
  12. 2024CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentationcited by 14
22 more
  1. 2023Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Modelscited by 11
  2. 2025A Scalable Framework for Evaluating Health Language Modelscited by 11
  3. 2024Muffin or Chihuahua? Challenging Multimodal Large Language Models with Multipanel VQAcited by 9
  4. 2025Enhanced Large Language Models for Effective Screening of Depression and Anxietycited by 9
  5. 2024'Quis custodiet ipsos custodes?' Who will watch the watchmen? On Detecting AI-generated peer-reviewscited by 8
  6. 2024Harnessing Business and Media Insights with Large Language Modelscited by 7
  7. 2024Reward-RAG: Enhancing RAG with Reward Driven Supervisioncited by 7
  8. 2024Automating Customer Needs Analysis: A Comparative Study of Large Language Models in the Travel Industrycited by 6
  9. 2024Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summariescited by 6
  10. 2024Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Modelscited by 5
  11. 2024On Evaluating The Performance of Watermarked Machine-Generated Texts Under Adversarial Attackscited by 4
  12. 2025Evaluation Under Imperfect Benchmarks and Ratings: A Case Study in Text Simplificationcited by 4
  13. 2024LLM Augmentations to support Analytical Reasoning over Multiple Documentscited by 3
  14. 2025Empowering Economic Simulation for Massively Multiplayer Online Games through Generative Agent-Based Modelingcited by 3
  15. 2025Aligned Textual Scoring Rulescited by 2
  16. 2024Attaining Human`s Desirable Outcomes in Human-AI Interaction via Structural Causal Gamescited by 1
  17. 2025Bridging the Gap Between Multimodal Foundation Models and World Modelscited by 1
  18. 2026Utilizing Metadata for Better Retrieval-Augmented Generationcited by 1
  19. 2026Judging the Judges: Human Validation of Multi-LLM Evaluation for High-Quality K--12 Science Instructional Materialscited by 1
  20. 2026From "Help" to Helpful: A Hierarchical Assessment of LLMs in Mental e-Health Applicationscited by 1
  21. 2024CHARP: Conversation History AwaReness Probing for Knowledge-grounded Dialogue Systemscited by 0
  22. 2026From Feature-Based Models to Generative AI: Validity Evidence for Constructed Response Scoringcited by 0