# Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Composition

arXiv: https://arxiv.org/abs/2412.13612 · HTML: https://mlsanity.com/paper/2412.13612/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 18 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Builds on (most cited first, 3 of 3)

- [Do Language Models Know When They're Hallucinating References?](https://mlsanity.com/paper/2305.18248/) (2023)
- [Long2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall](https://mlsanity.com/paper/2410.23000/) (2024)
- [CHIME: LLM-Assisted Hierarchical Organization of Scientific Studies for Literature Review Support](https://mlsanity.com/paper/2407.16148/) (2024)

## Built on by (most cited first, 17 of 17)

- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [SurveyGen: Quality-Aware Scientific Survey Generation with Large Language Models](https://mlsanity.com/paper/2508.17647/) (2025)
- [Large Language Models Penetration in Scholarly Writing and Peer Review](https://mlsanity.com/paper/2502.11193/) (2025)
- [All That Glisters Is Not Gold: A Benchmark for Reference-Free Counterfactual Financial Misinformation Detection](https://mlsanity.com/paper/2601.04160/) (2026)
- [Do Deployment Constraints Make LLMs Hallucinate Citations? An Empirical Study across Four Models and Five Prompting Regimes](https://mlsanity.com/paper/2603.07287/) (2026)
- [JudgeBoard: Benchmarking and Enhancing Small Language Models for Reasoning Evaluation](https://mlsanity.com/paper/2511.15958/) (2025)
- [How Open Must Language Models be to Enable Reliable Scientific Inference?](https://mlsanity.com/paper/2603.26539/) (2026)
- [Can Large Language Models Forecast What Researchers Study Next?](https://mlsanity.com/paper/2609.00747/) (2026)
- [InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis](https://mlsanity.com/paper/2604.13201/) (2026)
- [An Artificial Intelligence Driven Semantic Similarity-Based Pipeline for Rapid Literature](https://mlsanity.com/paper/2509.15292/) (2025)
- [Leveraging Language Models and RAG for Efficient Knowledge Discovery in Clinical Environments](https://mlsanity.com/paper/2601.04209/) (2025)
- [How Much Structure Do LLMs Need? Evaluating LLMs for Bibliometric Cluster Description](https://mlsanity.com/paper/2605.24351/) (2026)
- [Do AI chatbots find what experts would? Effects of model, user role, and sample size on study retrieval for medical questions](https://mlsanity.com/paper/2608.13786/) (2026)
- [Can Scientific Claims Be Removed from Large Language Models? A Systematic Evaluation of Claim-Level Unlearning](https://mlsanity.com/paper/2608.20960/) (2026)
- [LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform](https://mlsanity.com/paper/2608.21374/) (2026)
- [SurveyAgent-HKA: A multi-agent framework for scientific survey generation with LLMs and human knowledge augmentation](https://mlsanity.com/paper/2609.05938/) (2026)
- [Errors of LLM-Assisted Literature Retrieval in Environmental Science: A Comparison Study of Abstract versus Full-text Based Prompts](https://mlsanity.com/paper/2610.05690/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have emerged as a potential solution to automate the complex processes involved in writing literature reviews, such as literature collection, organization, and summarization. However, it is yet unclear how good LLMs are at automating comprehensive and reliable literature reviews. This study introduces a framework to automatically evaluate the performance of LLMs in three key tasks of literature writing: reference generation, literature summary, and literature review composition. We introduce multidimensional evaluation metrics that assess the hallucination rates in generated references and measure the semantic coverage and factual consistency of the literature summaries and compositions against human-written counterparts. The experimental results reveal that even the most advanced models still generate hallucinated references, despite recent progress. Moreover, we observe that the performance of different models varies across disciplines when it comes to writing literature reviews. These findings highlight the need for further research and development to improve the reliability of LLMs in automating academic literature reviews.