# On the Limitations of Large Language Models (LLMs): False Attribution

arXiv: https://arxiv.org/abs/2404.04631 · HTML: https://mlsanity.com/paper/2404.04631/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 49 references that are in the map.
- First posted on arXiv 6 Apr 2024.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [Faithful to the Original: Fact Aware Neural Abstractive Summarization](https://mlsanity.com/paper/1711.04434/) (2017)
- [Learning Active Learning from Data](https://mlsanity.com/paper/1703.03365/) (2017)
- [A Survey of Active Learning for Natural Language Processing](https://mlsanity.com/paper/2210.10109/) (2022)
- [A Survey of AI-generated Text Forensic Systems: Detection, Attribution, and Characterization](https://mlsanity.com/paper/2403.01152/) (2024)
- [Comparing Hallucination Detection Metrics for Multilingual Generation](https://mlsanity.com/paper/2402.10496/) (2024)
- [Neural Authorship Attribution: Stylometric Analysis on Large Language Models](https://mlsanity.com/paper/2308.07305/) (2023)
- [Generative AI and Teachers -- For Us or Against Us? A Case Study](https://mlsanity.com/paper/2404.03486/) (2024)
- [ProCoT: Stimulating Critical Thinking and Writing of Students through Engagement with Large Language Models (LLMs)](https://mlsanity.com/paper/2312.09801/) (2023)
- [T5 meets Tybalt: Author Attribution in Early Modern English Drama Using Large Language Models](https://mlsanity.com/paper/2310.18454/) (2023)
- [Instruction Makes a Difference](https://mlsanity.com/paper/2402.00453/) (2024)

## Built on by (most cited first, 13 of 13)

- [Fairness and Bias in Multimodal AI: A Survey](https://mlsanity.com/paper/2406.19097/) (2024)
- [Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judge](https://mlsanity.com/paper/2602.09341/) (2026)
- [LLMs Working in Harmony: A Survey on the Technological Aspects of Building Effective LLM-Based Multi Agent Systems](https://mlsanity.com/paper/2504.01963/) (2025)
- [Map&Make: Schema Guided Text to Table Generation](https://mlsanity.com/paper/2505.23174/) (2025)
- [Enhancing Large Language Models through Neuro-Symbolic Integration and Ontological Reasoning](https://mlsanity.com/paper/2504.07640/) (2025)
- [Trends and Challenges in Authorship Analysis: A Review of ML, DL, and LLM Approaches](https://mlsanity.com/paper/2505.15422/) (2025)
- [AI Must not be Fully Autonomous](https://mlsanity.com/paper/2507.23330/) (2025)
- [Document Attribution: Examining Citation Relationships using Large Language Models](https://mlsanity.com/paper/2505.06324/) (2025)
- [From the Rock Floor to the Cloud: A Systematic Survey of State-of-the-Art NLP in Battery Life Cycle](https://mlsanity.com/paper/2510.27369/) (2025)
- [BatteryPass-12K: The First Dataset for the Novel Digital Battery Passport Conformance Task](https://mlsanity.com/paper/2604.26986/) (2026)
- [Counterargument for Critical Thinking as Judged by AI and Humans](https://mlsanity.com/paper/2605.05353/) (2026)
- [Per-Entity Bias Mapping for AI Visibility: Why Brand Mentions Require Entity-Specific Calibration](https://mlsanity.com/paper/2606.21595/) (2026)
- [Author Representation Strategies for Zero-Shot Authorship Attribution: A Comparative Study of LLM-Based and Embedding-Based Approaches](https://mlsanity.com/paper/2610.03531/) (2026)

## Abstract (arXiv)

In this work, we introduce a new hallucination metric - Simple Hallucination Index (SHI) and provide insight into one important limitation of the parametric knowledge of large language models (LLMs), i.e. false attribution. The task of automatic author attribution for relatively small chunks of text is an important NLP task but can be challenging. We empirically evaluate the power of 3 open SotA LLMs in zero-shot setting (Gemma-7B, Mixtral 8x7B, and LLaMA-2-13B). We acquired the top 10 most popular books of a month, according to Project Gutenberg, divided each one into equal chunks of 400 words, and prompted each LLM to predict the author. We then randomly sampled 162 chunks per book for human evaluation, based on the error margin of 7% and a confidence level of 95%. The average results show that Mixtral 8x7B has the highest prediction accuracy, the lowest SHI, and a Pearson's correlation (r) of 0.724, 0.263, and -0.9996, respectively, followed by LLaMA-2-13B and Gemma-7B. However, Mixtral 8x7B suffers from high hallucinations for 3 books, rising as high as a SHI of 0.87 (in the range 0-1, where 1 is the worst). The strong negative correlation of accuracy and SHI, given by r, demonstrates the fidelity of the new hallucination metric, which may generalize to other tasks. We also show that prediction accuracies correlate positively with the frequencies of Wikipedia instances of the book titles instead of the downloads and we perform error analyses of predictions. We publicly release the annotated chunks of data and our codes to aid the reproducibility and evaluation of other models.