# Over-Optimization of Academic Publishing Metrics: Observing Goodhart's Law in Action

arXiv: https://arxiv.org/abs/1809.07841 · HTML: https://mlsanity.com/paper/1809.07841/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2018.

## Built on by (most cited first, 22 of 22)

- [ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models](https://mlsanity.com/paper/2404.07738/) (2024)
- [PaperQA: Retrieval-Augmented Generative Agent for Scientific Research](https://mlsanity.com/paper/2312.07559/) (2023)
- [PeerQA: A Scientific Question Answering Dataset from Peer Reviews](https://mlsanity.com/paper/2502.13668/) (2025)
- [IRIS: Interactive Research Ideation System for Accelerating Scientific Discovery](https://mlsanity.com/paper/2504.16728/) (2025)
- [LLMs as Meta-Reviewers' Assistants: A Case Study](https://mlsanity.com/paper/2402.15589/) (2024)
- [The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors](https://mlsanity.com/paper/2509.04484/) (2025)
- [One-Shot Strategic Classification Under Unknown Costs](https://mlsanity.com/paper/2311.02761/) (2023)
- [Hierarchical Catalogue Generation for Literature Review: A Benchmark](https://mlsanity.com/paper/2304.03512/) (2023)
- [A Review of the Evidence for Existential Risk from AI via Misaligned Power-Seeking](https://mlsanity.com/paper/2310.18244/) (2023)
- [Tight Lower Bounds and Improved Convergence in Performative Prediction](https://mlsanity.com/paper/2412.03671/) (2024)
- [Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks](https://mlsanity.com/paper/2603.04459/) (2026)
- [Enhancing literature review with LLM and NLP methods. Algorithmic trading case](https://mlsanity.com/paper/2411.05013/) (2024)
- [pAI/MSc: ML Theory Research with Humans on the Loop](https://mlsanity.com/paper/2604.20622/) (2026)
- [Graph2Idea:Retrieval-Augmented Scientific Idea Generation with Graph-Structured Contexts](https://mlsanity.com/paper/2606.09105/) (2026)
- [EvoGens: A Population-Based Heuristic Search Framework for Scientific Idea Generation](https://mlsanity.com/paper/2605.30961/) (2026)
- [The History of AI Rights Research](https://mlsanity.com/paper/2208.04714/) (2022)
- [Investigating Deep-Learning NLP for Automating the Extraction of Oncology Efficacy Endpoints from Scientific Literature](https://mlsanity.com/paper/2311.04925/) (2023)
- [SyROCCo: Enhancing Systematic Reviews using Machine Learning](https://mlsanity.com/paper/2406.16527/) (2024)
- [Where Do Your Citations Come From? Citation-Constellation: A Free, Open-Source, No-Code, and Auditable Tool for Citation Network Decomposition with Complementary BARON and HEROCON Scores](https://mlsanity.com/paper/2603.24216/) (2026)
- [LLM-Metrics: Measuring Research Impact Through Large Language Model Memory](https://mlsanity.com/paper/2605.22176/) (2026)
- [The Evolutionary Origin of Values: implications for AI alignment, sentience and existential risk](https://mlsanity.com/paper/2608.03361/) (2026)
- [Ethical LLM-Assisted Research: A Framework for Responsible Delegation, Verification, and Epistemic Value](https://mlsanity.com/paper/2608.23644/) (2026)

## Abstract (arXiv)

The academic publishing world is changing significantly, with ever-growing numbers of publications each year and shifting publishing patterns. However, the metrics used to measure academic success, such as the number of publications, citation number, and impact factor, have not changed for decades. Moreover, recent studies indicate that these metrics have become targets and follow Goodhart's Law, according to which "when a measure becomes a target, it ceases to be a good measure." In this study, we analyzed over 120 million papers to examine how the academic publishing world has evolved over the last century. Our study shows that the validity of citation-based measures is being compromised and their usefulness is lessening. In particular, the number of publications has ceased to be a good metric as a result of longer author lists, shorter papers, and surging publication numbers. Citation-based metrics, such citation number and h-index, are likewise affected by the flood of papers, self-citations, and lengthy reference lists. Measures such as a journal's impact factor have also ceased to be good metrics due to the soaring numbers of papers that are published in top journals, particularly from the same pool of authors. Moreover, by analyzing properties of over 2600 research fields, we observed that citation-based metrics are not beneficial for comparing researchers in different fields, or even in the same department. Academic publishing has changed considerably; now we need to reconsider how we measure success.