# The Generative AI Paradox on Evaluation: What It Can Solve, It May Not Evaluate

arXiv: https://arxiv.org/abs/2402.06204 · HTML: https://mlsanity.com/paper/2402.06204/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 9 Feb 2024.

## Builds on (most cited first, 12 of 12)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [The Generative AI Paradox: "What It Can Create, It May Not Understand"](https://mlsanity.com/paper/2311.00059/) (2023)

## Built on by (most cited first, 16 of 16)

- [Small Language Models Need Strong Verifiers to Self-Correct Reasoning](https://mlsanity.com/paper/2404.17140/) (2024)
- [Benchmarks as Microscopes: A Call for Model Metrology](https://mlsanity.com/paper/2407.16711/) (2024)
- [The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?](https://mlsanity.com/paper/2402.19475/) (2024)
- [Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks](https://mlsanity.com/paper/2510.01232/) (2025)
- [Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice](https://mlsanity.com/paper/2409.07713/) (2024)
- [A Principle-based Framework for the Development and Evaluation of Large Language Models for Health and Wellness](https://mlsanity.com/paper/2512.08936/) (2025)
- [Over the Edge of Chaos? Excess Complexity as a Roadblock to Artificial General Intelligence](https://mlsanity.com/paper/2407.03652/) (2024)
- [Uncovering Factor Level Preferences to Improve Human-Model Alignment](https://mlsanity.com/paper/2410.06965/) (2024)
- [Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corpora](https://mlsanity.com/paper/2505.08905/) (2025)
- [Generative-Evaluative Agreement: A Necessary Validity Criterion for LLM-Enabled Adaptive Assessment](https://mlsanity.com/paper/2605.19529/) (2026)
- [Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA](https://mlsanity.com/paper/2606.28050/) (2026)
- [Concepts Learned Visually by Infants Can Contribute to Visual Learning and Understanding in AI Models](https://mlsanity.com/paper/2503.03361/) (2025)
- [LLMs Are Not Scorers: Rethinking MT Evaluation with Generation-Based Methods](https://mlsanity.com/paper/2505.16129/) (2025)
- [CLINB: A Climate Intelligence Benchmark for Foundational Models](https://mlsanity.com/paper/2511.11597/) (2025)
- [The Metanym Game: An LLM Benchmark Without Ground Truth That Rises With the Models It Measures](https://mlsanity.com/paper/2606.21008/) (2026)
- [Training Therapeutic Judges and Multi-Agent Systems for Human-Aligned Mental Health Support](https://mlsanity.com/paper/2606.30887/) (2026)

## Abstract (arXiv)

This paper explores the assumption that Large Language Models (LLMs) skilled in generation tasks are equally adept as evaluators. We assess the performance of three LLMs and one open-source LM in Question-Answering (QA) and evaluation tasks using the TriviaQA (Joshi et al., 2017) dataset. Results indicate a significant disparity, with LLMs exhibiting lower performance in evaluation tasks compared to generation tasks. Intriguingly, we discover instances of unfaithful evaluation where models accurately evaluate answers in areas where they lack competence, underscoring the need to examine the faithfulness and trustworthiness of LLMs as evaluators. This study contributes to the understanding of "the Generative AI Paradox" (West et al., 2023), highlighting a need to explore the correlation between generative excellence and evaluation proficiency, and the necessity to scrutinize the faithfulness aspect in model evaluations.