mlsanityData to 7 Oct 2026

Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?

arXiv 2302.03495 ↗First posted 3 Feb 2023cs.IR cs.AIcited by 54

Not on the map. Why?

Abstract (arXiv)

Systematic reviews are comprehensive reviews of the literature for a highly focused research question. These reviews are often treated as the highest form of evidence in evidence-based medicine, and are the key strategy to answer research questions in the medical field. To create a high-quality systematic review, complex Boolean queries are often constructed to retrieve studies for the review topic. However, it often takes a long time for systematic review researchers to construct a high quality systematic review Boolean query, and often the resulting queries are far from effective. Poor queries may lead to biased or invalid reviews, because they missed to retrieve key evidence, or to extensive increase in review costs, because they retrieved too many irrelevant studies. Recent advances in Transformer-based generative models have shown great potential to effectively follow instructions from users and generate answers based on the instructions being made. In this paper, we investigate the effectiveness of the latest of such models, ChatGPT, in generating effective Boolean queries for systematic review literature search. Through a number of extensive experiments on standard test collections for the task, we find that ChatGPT is capable of generating queries that lead to high search precision, although trading-off this for recall. Overall, our study demonstrates the potential of ChatGPT in generating effective Boolean queries for systematic review literature search. The ability of ChatGPT to follow complex instructions and generate queries with high precision makes it a valuable tool for researchers conducting systematic reviews, particularly for rapid reviews where time is a constraint and often trading-off higher precision for lower recall is acceptable.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 14 of its 87 references that are in the map. First posted on arXiv 3 Feb 2023.

Builds on

14 of 87 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  4. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  5. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  6. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  7. 2019Multi-Stage Document Ranking with BERTcited by 200
  8. 2022Can large language models reason about medical questions?cited by 194
  9. 2023ChatGPT is not all you need. A State of the Art Review of large Generative AI modelscited by 85
  10. 2022The Creativity of Text-to-Image Generationcited by 73
  11. 2022A Taxonomy of Prompt Modifiers for Text-To-Image Generationcited by 54
  12. 2022From Little Things Big Things Grow: A Collection with Seed Studies for Medical Systematic Review Literature Searchcited by 10
2 more
  1. 2021Search Strategy Formulation for Systematic Reviews: issues, challenges and opportunitiescited by 6
  2. 2022Automated MeSH Term Suggestion for Effective Query Formulation in Systematic Reviews Literature Searchcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 23; 2024: 16; 2025: 8; 2026: 7.

54 papers.

Sort:

By area:

  1. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  2. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  3. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  4. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  5. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  6. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  7. 2023Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Familycited by 71
  8. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  9. 2023One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Eracited by 55
  10. 2023A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Modelscited by 53
  11. 2024Measuring and Controlling Instruction (In)Stability in Language Model Dialogscited by 51
  12. 2023Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potentialcited by 39
38 more
  1. 2023Can large language models replace humans in the systematic review process? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languagescited by 36
  2. 2024AI for social science and social science of AI: A Surveycited by 29
  3. 2024The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic reviewcited by 20
  4. 2023Search Still Matters: Information Retrieval in the Era of Generative AIcited by 17
  5. 2025Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performancecited by 17
  6. 2023Does ChatGPT have Theory of Mind?cited by 16
  7. 2023PubMed and Beyond: Biomedical Literature Search in the Age of Artificial Intelligencecited by 16
  8. 2024Accelerating Clinical Evidence Synthesis with Large Language Modelscited by 15
  9. 2023ChatGPT Hallucinates when Attributing Answerscited by 14
  10. 2024Zero-shot Generative Large Language Models for Systematic Review Screening Automationcited by 14
  11. 2023Unlocking the Potential of ChatGPT: A Comprehensive Exploration of its Applications, Advantages, Limitations, and Future Directions in Natural Language Processingcited by 11
  12. 2025A foundation model for human-AI collaboration in medical literature miningcited by 11
  13. 2023Assessing the Ability of ChatGPT to Screen Articles for Systematic Reviewscited by 10
  14. 2024DiscipLink: Unfolding Interdisciplinary Information Seeking Process via Human-AI Co-Explorationcited by 10
  15. 2024Synthetic Query Generation using Large Language Models for Virtual Assistantscited by 9
  16. 2024A Perspective for Adapting Generalist AI to Specialized Medical AI Applications and Their Challengescited by 8
  17. 2024HumanEval on Latest GPT Models -- 2024cited by 7
  18. 2024An Investigation of Prompt Variations for Zero-shot LLM-based Rankerscited by 7
  19. 2023Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automationcited by 6
  20. 2023Generating Natural Language Queries for More Effective Systematic Review Screening Prioritisationcited by 6
  21. 2024Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detectioncited by 6
  22. 2025Precise Zero-Shot Pointwise Ranking with LLMs through Post-Aggregated Global Context Informationcited by 6
  23. 2023Utilizing ChatGPT to Enhance Clinical Trial Enrollmentcited by 4
  24. 2023ChatGPT as a mapping assistant: A novel method to enrich maps with generative AI and content derived from street-level photographscited by 4
  25. 2025Reassessing Large Language Model Boolean Query Generation for Systematic Reviewscited by 4
  26. 2026Search, Inspect, Fetch: Exploiting Structure-Aware Boolean Retrieval for Deep-Search Agentscited by 3
  27. 2023CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviewscited by 2
  28. 2024Exploring ChatGPT for Next-generation Information Retrieval: Opportunities and Challengescited by 2
  29. 2025Are Human Interactions Replicable by Generative Agents? A Case Study on Pronoun Usage in Hierarchical Interactionscited by 2
  30. 2025Conversational Exploration of Literature Landscape with LitChatcited by 2
  31. 2024Large Language Models for Stemming: Promises, Pitfalls and Failurescited by 1
  32. 2024On the Decision-Making Abilities in Role-Playing using Large Language Modelscited by 1
  33. 2025Large language models streamline automated systematic review: A preliminary studycited by 1
  34. 2023ChatGPT-4 as a Tool for Reviewing Academic Books in Spanishcited by 0
  35. 2024Bottom-up Anytime Discovery of Generalised Multimodal Graph Patterns for Knowledge Graphscited by 0
  36. 2025An Artificial Intelligence Driven Semantic Similarity-Based Pipeline for Rapid Literaturecited by 0
  37. 2026AutoScreen-FW: An LLM-based Framework for Resume Screeningcited by 0
  38. 2026On-Demand Instructional Material Providing Agent Based on MLLM for Tutoring Supportcited by 0

50 most cited of 54 All 54 (JSON)