# Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4 Text Ratings

arXiv: https://arxiv.org/abs/2308.02575 · HTML: https://mlsanity.com/paper/2308.02575/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 34 references that are in the map.
- First posted on arXiv 3 Aug 2023.

## Builds on (most cited first, 14 of 14)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [Prompting GPT-3 To Be Reliable](https://mlsanity.com/paper/2210.09150/) (2022)
- [A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT](https://mlsanity.com/paper/2303.04226/) (2023)
- [Boosting Theory-of-Mind Performance in Large Language Models via Prompting](https://mlsanity.com/paper/2304.11490/) (2023)
- [TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks](https://mlsanity.com/paper/2305.11430/) (2023)
- [Learning to Perform Complex Tasks through Compositional Fine-Tuning of Language Models](https://mlsanity.com/paper/2210.12607/) (2022)

## Built on by (most cited first, 30 of 34)

- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [StruQ: Defending Against Prompt Injection with Structured Queries](https://mlsanity.com/paper/2402.06363/) (2024)
- [Orca 2: Teaching Small Language Models How to Reason](https://mlsanity.com/paper/2311.11045/) (2023)
- [Jatmo: Prompt Injection Defense by Task-Specific Finetuning](https://mlsanity.com/paper/2312.17673/) (2023)
- [Mark My Words: Analyzing and Evaluating Language Model Watermarks](https://mlsanity.com/paper/2312.00273/) (2023)
- [MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos](https://mlsanity.com/paper/2406.08407/) (2024)
- [HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics](https://mlsanity.com/paper/2410.09988/) (2024)
- [How Benchmark Prediction from Fewer Data Misses the Mark](https://mlsanity.com/paper/2506.07673/) (2025)
- [Reinforcement Learning for Optimizing RAG for Domain Chatbots](https://mlsanity.com/paper/2401.06800/) (2024)
- [Can AI grade your essays? A comparative analysis of large language models and teacher ratings in multidimensional essay scoring](https://mlsanity.com/paper/2411.16337/) (2024)
- [WeaverBird: Empowering Financial Decision-Making with Large Language Model, Knowledge Base, and Search Engine](https://mlsanity.com/paper/2308.05361/) (2023)
- [CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation](https://mlsanity.com/paper/2409.02098/) (2024)
- [Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models](https://mlsanity.com/paper/2310.07611/) (2023)
- [A Scalable Framework for Evaluating Health Language Models](https://mlsanity.com/paper/2503.23339/) (2025)
- [Muffin or Chihuahua? Challenging Multimodal Large Language Models with Multipanel VQA](https://mlsanity.com/paper/2401.15847/) (2024)
- [Enhanced Large Language Models for Effective Screening of Depression and Anxiety](https://mlsanity.com/paper/2501.08769/) (2025)
- ['Quis custodiet ipsos custodes?' Who will watch the watchmen? On Detecting AI-generated peer-reviews](https://mlsanity.com/paper/2410.09770/) (2024)
- [Harnessing Business and Media Insights with Large Language Models](https://mlsanity.com/paper/2406.06559/) (2024)
- [Reward-RAG: Enhancing RAG with Reward Driven Supervision](https://mlsanity.com/paper/2410.03780/) (2024)
- [Automating Customer Needs Analysis: A Comparative Study of Large Language Models in the Travel Industry](https://mlsanity.com/paper/2404.17975/) (2024)
- [Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries](https://mlsanity.com/paper/2409.00844/) (2024)
- [Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Models](https://mlsanity.com/paper/2405.12939/) (2024)
- [On Evaluating The Performance of Watermarked Machine-Generated Texts Under Adversarial Attacks](https://mlsanity.com/paper/2407.04794/) (2024)
- [Evaluation Under Imperfect Benchmarks and Ratings: A Case Study in Text Simplification](https://mlsanity.com/paper/2504.09394/) (2025)
- [LLM Augmentations to support Analytical Reasoning over Multiple Documents](https://mlsanity.com/paper/2411.16116/) (2024)
- [Empowering Economic Simulation for Massively Multiplayer Online Games through Generative Agent-Based Modeling](https://mlsanity.com/paper/2506.04699/) (2025)
- [Aligned Textual Scoring Rules](https://mlsanity.com/paper/2507.06221/) (2025)
- [Attaining Human`s Desirable Outcomes in Human-AI Interaction via Structural Causal Games](https://mlsanity.com/paper/2405.16588/) (2024)
- [Bridging the Gap Between Multimodal Foundation Models and World Models](https://mlsanity.com/paper/2510.03727/) (2025)
- [Utilizing Metadata for Better Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.11863/) (2026)

## Abstract (arXiv)

This study investigates the consistency of feedback ratings generated by OpenAI's GPT-4, a state-of-the-art artificial intelligence language model, across multiple iterations, time spans and stylistic variations. The model rated responses to tasks within the Higher Education (HE) subject domain of macroeconomics in terms of their content and style. Statistical analysis was conducted in order to learn more about the interrater reliability, consistency of the ratings across iterations and the correlation between ratings in terms of content and style. The results revealed a high interrater reliability with ICC scores ranging between 0.94 and 0.99 for different timespans, suggesting that GPT-4 is capable of generating consistent ratings across repetitions with a clear prompt. Style and content ratings show a high correlation of 0.87. When applying a non-adequate style the average content ratings remained constant, while style ratings decreased, which indicates that the large language model (LLM) effectively distinguishes between these two criteria during evaluation. The prompt used in this study is furthermore presented and explained. Further research is necessary to assess the robustness and reliability of AI models in various use cases.