mlsanityData to 14 Oct 2026

LLMs for Academic Workflows: An Evaluation of Literature Reviews Generated with Short and Long Context Windows of LLMs

arXiv 2608.26145 ↗First posted 28 Jun 2026cs.AI cs.HC cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

Our research focuses on evaluating literature reviews generated in short and long context settings of large language models (LLMs) to investigate the impact of context window on the quality of AI-generated literature reviews and the role of AI in supporting literature review writing. Twenty AI-generated literature reviews based on research sources from Semantic Scholar and Arxiv were evaluated by two researchers across 15 dimensions. Our findings reveal that AI-generated literature reviews require human oversight to meet academic publishing standards. As context windows increase, LLMs can incorporate broader information and maintain coherence across longer inputs, but they also exacerbate issues such as content repetition, omission of critical work, and a tendency towards descriptiveness over synthesis. Our work shows that AI-generated reviews can provide foundational overviews, but their output must be critically evaluated and refined by domain experts. Future research should consider integrating other LLMs and fine-tuned models in different domains with hybrid approaches that combine human expertise with AI capabilities to address the limitations identified in this study.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 24 references that are in the map. First posted on arXiv 28 Jun 2026.

Builds on

14 of 24 references.

References · 14

  1. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  2. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  3. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  4. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  5. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  6. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  7. 2024On Limitations of the Transformer Architecturecited by 63
  8. 2023Extending Context Window of Large Language Models via Semantic Compressioncited by 45
  9. 2024AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviewscited by 44
  10. 2024CHIME: LLM-Assisted Hierarchical Organization of Scientific Studies for Literature Review Supportcited by 29
  11. 2024Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlesslycited by 24
  12. 2024Extending LLMs' Context Window with 100 Samplescited by 21
2 more
  1. 2023Augmenting LLMs with Knowledge: A survey on hallucination preventioncited by 16
  2. 2024Systematic Review on Healthcare Systems Engineering utilizing ChatGPTcited by 2

Built on by

Not cited yet.