# LitLLMs, LLMs for Literature Review: Are we there yet?

arXiv: https://arxiv.org/abs/2412.15249 · HTML: https://mlsanity.com/paper/2412.15249/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 82 references that are in the map.
- First posted on arXiv 15 Dec 2024.

## Builds on (most cited first, 30 of 62)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models](https://mlsanity.com/paper/2305.04091/) (2023)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models](https://mlsanity.com/paper/2212.04088/) (2022)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://mlsanity.com/paper/2304.09542/) (2023)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)
- [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://mlsanity.com/paper/2112.07916/) (2021)

## Built on by (most cited first, 14 of 14)

- [MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research](https://mlsanity.com/paper/2505.19955/) (2025)
- [ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review](https://mlsanity.com/paper/2510.08867/) (2025)
- [Benchmarking AI Agents for Addressing Scientific Challenges Across Scales](https://mlsanity.com/paper/2606.12736/) (2026)
- [Position: The ML Community Must Build an AI-Augmented Peer-Review Ecosystem](https://mlsanity.com/paper/2506.08134/) (2025)
- [Attribution, Citation, and Quotation: A Survey of Evidence-based Text Generation with Large Language Models](https://mlsanity.com/paper/2508.15396/) (2025)
- [Modelling and Classifying the Components of a Literature Review](https://mlsanity.com/paper/2508.04337/) (2025)
- [IV Co-Scientist: Multi-Agent LLM Framework for Causal Instrumental Variable Discovery](https://mlsanity.com/paper/2602.07943/) (2026)
- [No Hidden Prompts Needed! You Can Game AI Peer Review with Presentation-Only Revisions](https://mlsanity.com/paper/2606.13044/) (2026)
- [Accelerating Scientific Research with Gemini in the Real-World](https://mlsanity.com/paper/2608.26701/) (2026)
- [Arti-"fickle" Intelligence: Using LLMs as a Tool for Inference in the Political and Social Sciences](https://mlsanity.com/paper/2504.03822/) (2025)
- [Accelerating Scientific Discovery with Multi-Document Summarization of Impact-Ranked Papers](https://mlsanity.com/paper/2508.03962/) (2025)
- [Meow: End-to-End Outline Writing for Automatic Academic Survey](https://mlsanity.com/paper/2509.19370/) (2025)
- [TCMIIES: A Browser-Based LLM-Powered Intelligent Information Extraction System for Academic Literature](https://mlsanity.com/paper/2605.07507/) (2026)
- [Can LLMs extract scientific consensus? A case study in high-temperature superconductivity](https://mlsanity.com/paper/2606.07570/) (2026)

## Abstract (arXiv)

Literature reviews are an essential component of scientific research, but they remain time-intensive and challenging to write, especially due to the recent influx of research papers. This paper explores the zero-shot abilities of recent Large Language Models (LLMs) in assisting with the writing of literature reviews based on an abstract. We decompose the task into two components: 1. Retrieving related works given a query abstract, and 2. Writing a literature review based on the retrieved results. We analyze how effective LLMs are for both components. For retrieval, we introduce a novel two-step search strategy that first uses an LLM to extract meaningful keywords from the abstract of a paper and then retrieves potentially relevant papers by querying an external knowledge base. Additionally, we study a prompting-based re-ranking mechanism with attribution and show that re-ranking doubles the normalized recall compared to naive search methods, while providing insights into the LLM's decision-making process. In the generation phase, we propose a two-step approach that first outlines a plan for the review and then executes steps in the plan to generate the actual review. To evaluate different LLM-based literature review methods, we create test sets from arXiv papers using a protocol designed for rolling use with newly released LLMs to avoid test set contamination in zero-shot evaluations. We release this evaluation protocol to promote additional research and development in this regard. Our empirical results suggest that LLMs show promising potential for writing literature reviews when the task is decomposed into smaller components of retrieval and planning. Our project page including a demonstration system and toolkit can be accessed here: https://litllm.github.io.