Errors of LLM-Assisted Literature Retrieval in Environmental Science: A Comparison Study of Abstract versus Full-text Based Prompts
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) are increasingly used for literature search and synthesis. However, it is unclear whether they retrieve accurate bibliographic information in environmental science. Therefore, we quantitatively compared the errors of widely used LLM platforms in retrieving references related to original articles from five leading environmental science journals (Energy and Environmental Science, Nature Sustainability, Nature Climate Change, Lancet Planetary Health, and Environmental Science and Technology) published in 2024 to 2025. Claude, ChatGPT, Grok, DeepSeek, Perplexity, and Gemini were used as the LLM platforms. LLMs retrieved 10 references for each of the 50 randomly selected original article using either the article's abstract or its full-text as prompt. The retrieved references were subject to a multimetric score ratio combining validity of bibliographic data, Google Scholar link, digital object identifier, Scopus Electronic Identifier and relevance score (cited by or being the index paper), and the proportion of complete fabrication that failed all metrics. Abstract-only prompt yielded significantly higher accuracy than full-text one. This advantage was confirmed in multilevel mixed-effect multivariable regression after adjusting for journal, platform, and output order. Source journal and the position of a reference within the output list were also independently associated with retrieval accuracy, with lower-listed references associated with lower accuracy. These findings suggest that LLM assisted literature retrieval in environmental science remains moderately accurate and overall inconsistent, varying significantly by platform, journal, prompt type, and output position. Abstract-based prompting, as task-aligned information compression, may outperform full-text one in literature retrieval. Caution should be used when generalizing our findings.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 33 references that are in the map. First posted on arXiv 5 Oct 2026.
Builds on
13 of 33 references.
References · 13
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
- 2023Evaluating Verifiability in Generative Search Enginescited by 303
- 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
- 2024Concise Thoughts: Impact of Output Length on LLM Reasoning and Costcited by 109
- 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inferencecited by 53
- 2024The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic reviewcited by 21
- 2024Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Compositioncited by 17
- 2023Improving accuracy of GPT-3/4 results on biomedical data using a retrieval-augmented language modelcited by 15
- 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inferencecited by 9
- 2026RAGRouter-Bench: A Dataset and Benchmark for Adaptive RAG Routingcited by 6
- 2025Trustworthy AI for Medicine: Continuous Hallucination Detection and Elimination with CHECKcited by 5
1 more
Built on by
Not cited yet.