# Highlighting Case Studies in LLM Literature Review of Interdisciplinary System Science

arXiv: https://arxiv.org/abs/2503.16515 · HTML: https://mlsanity.com/paper/2503.16515/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 40 references that are in the map.
- First posted on arXiv 16 Mar 2025.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Changing the World by Changing the Data](https://mlsanity.com/paper/2105.13947/) (2021)
- [Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3 (with Varying Success)](https://mlsanity.com/paper/2305.06299/) (2023)
- [ChatCite: LLM Agent with Human Workflow Guidance for Comparative Literature Summary](https://mlsanity.com/paper/2403.02574/) (2024)
- [Nuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Prompting](https://mlsanity.com/paper/2304.11686/) (2023)
- [System for systematic literature review using multiple AI agents: Concept and an empirical evaluation](https://mlsanity.com/paper/2403.08399/) (2024)
- [Deriving Contextualised Semantic Features from BERT (and Other Transformer Model) Embeddings](https://mlsanity.com/paper/2012.15353/) (2020)
- [Divide and Conquer: Text Semantic Matching with Disentangled Keywords and Intents](https://mlsanity.com/paper/2203.02898/) (2022)
- [Automated Theorem Provers Help Improve Large Language Model Reasoning](https://mlsanity.com/paper/2408.03492/) (2024)
- [AI Insights: A Case Study on Utilizing ChatGPT Intelligence for Research Paper Analysis](https://mlsanity.com/paper/2403.03293/) (2024)

## Built on by (most cited first, 1 of 1)

- [Overview of AI Grading of Physics Olympiad Exams](https://mlsanity.com/paper/2505.02121/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) were used to assist four Commonwealth Scientific and Industrial Research Organisation (CSIRO) researchers to perform systematic literature reviews (SLR). We evaluate the performance of LLMs for SLR tasks in these case studies. In each, we explore the impact of changing parameters on the accuracy of LLM responses. The LLM was tasked with extracting evidence from chosen academic papers to answer specific research questions. We evaluate the models' performance in faithfully reproducing quotes from the literature and subject experts were asked to assess the model performance in answering the research questions. We developed a semantic text highlighting tool to facilitate expert review of LLM responses. We found that state of the art LLMs were able to reproduce quotes from texts with greater than 95% accuracy and answer research questions with an accuracy of approximately 83%. We use two methods to determine the correctness of LLM responses; expert review and the cosine similarity of transformer embeddings of LLM and expert answers. The correlation between these methods ranged from 0.48 to 0.77, providing evidence that the latter is a valid metric for measuring semantic similarity.