# LLMs for Academic Workflows: An Evaluation of Literature Reviews Generated with Short and Long Context Windows of LLMs

arXiv: https://arxiv.org/abs/2608.26145 · HTML: https://mlsanity.com/paper/2608.26145/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 28 Jun 2026.

## Builds on (most cited first, 14 of 14)

- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection](https://mlsanity.com/paper/2403.03507/) (2024)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [BooookScore: A systematic exploration of book-length summarization in the era of LLMs](https://mlsanity.com/paper/2310.00785/) (2023)
- [PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training](https://mlsanity.com/paper/2309.10400/) (2023)
- [Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2311.12351/) (2023)
- [On Limitations of the Transformer Architecture](https://mlsanity.com/paper/2402.08164/) (2024)
- [Extending Context Window of Large Language Models via Semantic Compression](https://mlsanity.com/paper/2312.09571/) (2023)
- [AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviews](https://mlsanity.com/paper/2408.10365/) (2024)
- [CHIME: LLM-Assisted Hierarchical Organization of Scientific Studies for Literature Review Support](https://mlsanity.com/paper/2407.16148/) (2024)
- [Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly](https://mlsanity.com/paper/2408.01866/) (2024)
- [Extending LLMs' Context Window with 100 Samples](https://mlsanity.com/paper/2401.07004/) (2024)
- [Augmenting LLMs with Knowledge: A survey on hallucination prevention](https://mlsanity.com/paper/2309.16459/) (2023)
- [Systematic Review on Healthcare Systems Engineering utilizing ChatGPT](https://mlsanity.com/paper/2405.11817/) (2024)

## Abstract (arXiv)

Our research focuses on evaluating literature reviews generated in short and long context settings of large language models (LLMs) to investigate the impact of context window on the quality of AI-generated literature reviews and the role of AI in supporting literature review writing. Twenty AI-generated literature reviews based on research sources from Semantic Scholar and Arxiv were evaluated by two researchers across 15 dimensions. Our findings reveal that AI-generated literature reviews require human oversight to meet academic publishing standards. As context windows increase, LLMs can incorporate broader information and maintain coherence across longer inputs, but they also exacerbate issues such as content repetition, omission of critical work, and a tendency towards descriptiveness over synthesis. Our work shows that AI-generated reviews can provide foundational overviews, but their output must be critically evaluated and refined by domain experts. Future research should consider integrating other LLMs and fine-tuned models in different domains with hybrid approaches that combine human expertise with AI capabilities to address the limitations identified in this study.