Hierarchical Memorization in Large Language Models: Evidence from Citation Generation
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) generate fluent text across a wide range of tasks, but the fabrication of non-existent academic citations remains a critical and well-documented failure mode. Building on prior work that frames hallucination and verbatim memorization as outcomes of the same probabilistic process, this study uses citation count as a proxy for training data redundancy and asks how this redundancy is internally structured within a single bibliographic record. Using GPT-4.1, we generated and manually verified 100 citations across twenty computer-science domains, measuring factual fidelity via cosine similarity against authentic metadata. We find that (i) factual accuracy varies substantially across domains and scales log-linearly with citation count, (ii) the model crosses two empirically identifiable thresholds; an inflection around 90 citations and a saturation point near 1,200 citations beyond which records are reproduced nearly verbatim, (iii) memorization is hierarchical, with titles and first authors recalled earliest while venues and numeric fields require far greater redundancy and publication years remain essentially unlearned, and (iv) even highly cited records can be conflated when their titles and authors overlap, an effect interpretable as spurious-attractor interference. Memorization in LLMs is therefore not a binary on/off state but a graduated, hierarchically layered phenomenon shaped by the uneven distribution of knowledge in the pretraining corpus.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 39 of its 70 references that are in the map. First posted on arXiv 12 Nov 2025.
Builds on
39 of 70 references.
References · 39
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2022Generative Adversarial Networkscited by 5,603
27 more
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2023A Survey of Large Language Modelscited by 2,537
- 2020Extracting Training Data from Large Language Modelscited by 2,144
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
- 2022Survey of Hallucination in Natural Language Generationcited by 2,056
- 2019Language Models as Knowledge Bases?cited by 1,922
- 2022Quantifying Memorization Across Neural Language Modelscited by 755
- 2021Revisiting Deep Learning Models for Tabular Datacited by 725
- 2021Deduplicating Training Data Makes Language Models Bettercited by 705
- 2020Self-supervised Learning: Generative or Contrastivecited by 693
- 2022Towards Reasoning in Large Language Models: A Surveycited by 593
- 2017Generative Adversarial Networks: An Overviewcited by 588
- 2020Hopfield Networks is All You Needcited by 461
- 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
- 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
- 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
- 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
- 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
- 2025Why Language Models Hallucinatecited by 205
- 2023Large Language Models as General Pattern Machinescited by 196
- 2022Mega: Moving Average Equipped Gated Attentioncited by 178
- 2024Chain of Thoughtlessness? An Analysis of CoT in Planningcited by 100
- 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 61
- 2020Citation Recommendation: Approaches and Datasetscited by 36
- 2019A Context-Aware Citation Recommendation Model with BERT and Graph Convolutional Networkscited by 28
- 2020Scientific Paper Recommendation: A Surveycited by 20
Built on by
Citing papers per year: 2026: 3.
3 papers.