# Large Language Models Reflect Human Citation Patterns with a Heightened Citation Bias

arXiv: https://arxiv.org/abs/2405.15739 · HTML: https://mlsanity.com/paper/2405.15739/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 38 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 12 of 12)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [Emergent autonomous scientific research capabilities of large language models](https://mlsanity.com/paper/2304.05332/) (2023)
- [The Semantic Scholar Open Data Platform](https://mlsanity.com/paper/2301.10140/) (2023)
- [Large Language Models for Scientific Synthesis, Inference and Explanation](https://mlsanity.com/paper/2310.07984/) (2023)
- [Social Bias Probing: Fairness Benchmarking for Language Models](https://mlsanity.com/paper/2311.09090/) (2023)
- [Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning](https://mlsanity.com/paper/2404.08680/) (2024)

## Built on by (most cited first, 17 of 17)

- [Scientific production in the era of Large Language Models](https://mlsanity.com/paper/2601.13187/) (2026)
- [How Deep Do Large Language Models Internalize Scientific Literature and Citation Practices?](https://mlsanity.com/paper/2504.02767/) (2025)
- [Proof of Time: A Benchmark for Evaluating Scientific Idea Judgments](https://mlsanity.com/paper/2601.07606/) (2026)
- [Who Gets Cited? Gender- and Majority-Bias in LLM-Driven Reference Selection](https://mlsanity.com/paper/2508.02740/) (2025)
- [Pre-review to Peer review: Pitfalls of Automating Reviews using Large Language Models](https://mlsanity.com/paper/2512.22145/) (2025)
- [Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agents](https://mlsanity.com/paper/2604.03173/) (2026)
- [Think Before Writing: Feature-Level Multi-Objective Optimization for Generative Citation Visibility](https://mlsanity.com/paper/2604.19113/) (2026)
- [Rethinking Evidence Hierarchies in Medical Language Benchmarks: A Critical Evaluation of HealthBench](https://mlsanity.com/paper/2508.00081/) (2025)
- [BibTeX Citation Errors in Scientific Publishing Agents: Evaluation and Mitigation](https://mlsanity.com/paper/2604.03159/) (2026)
- [Structurally Human, Semantically Biased: Detecting LLM-Generated References with Embeddings and GNNs](https://mlsanity.com/paper/2601.20704/) (2026)
- [Do AI chatbots find what experts would? Effects of model, user role, and sample size on study retrieval for medical questions](https://mlsanity.com/paper/2608.13786/) (2026)
- [On the Role of Citations in Preference Data](https://mlsanity.com/paper/2608.21376/) (2026)
- [Sycophants in the Courtroom: Are LLMs Fragile to Juridical Authority and Evolving Legal Standards?](https://mlsanity.com/paper/2608.21409/) (2026)
- [Authority Bias in Conversational Search Engines for Academic Paper Recommendation](https://mlsanity.com/paper/2609.00248/) (2026)
- [Citing Less Critically: LLMs Reshape the Rhetoric and Reach of Scientific Citation](https://mlsanity.com/paper/2609.01432/) (2026)
- [ReCite: Agentic Reasoning for Faithful Citation](https://mlsanity.com/paper/2609.09156/) (2026)
- [Big-Big-SycoBench: A Benchmark to Detect Sycophancy Through Model Responses to Valid vs. Invalid Arguments](https://mlsanity.com/paper/2610.15634/) (2026)

## Abstract (arXiv)

Citation practices are crucial in shaping the structure of scientific knowledge, yet they are often influenced by contemporary norms and biases. The emergence of Large Language Models (LLMs) introduces a new dynamic to these practices. Interestingly, the characteristics and potential biases of references recommended by LLMs that entirely rely on their parametric knowledge, and not on search or retrieval-augmented generation, remain unexplored. Here, we analyze these characteristics in an experiment using a dataset from AAAI, NeurIPS, ICML, and ICLR, published after GPT-4's knowledge cut-off date. In our experiment, LLMs are tasked with suggesting scholarly references for the anonymized in-text citations within these papers. Our findings reveal a remarkable similarity between human and LLM citation patterns, but with a more pronounced high citation bias, which persists even after controlling for publication year, title length, number of authors, and venue. The results hold for both GPT-4, and the more capable models GPT-4o and Claude 3.5 where the papers are part of the training data. Additionally, we observe a large consistency between the characteristics of LLM's existing and non-existent generated references, indicating the model's internalization of citation patterns. By analyzing citation graphs, we show that the references recommended are embedded in the relevant citation context, suggesting an even deeper conceptual internalization of the citation networks. While LLMs can aid in citation generation, they may also amplify existing biases, such as the Matthew effect, and introduce new ones, potentially skewing scientific knowledge dissemination.