# Empirical Evaluation of ChatGPT on Requirements Information Retrieval Under Zero-Shot Setting

arXiv: https://arxiv.org/abs/2304.12562 · HTML: https://mlsanity.com/paper/2304.12562/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 70 references that are in the map.
- First posted on arXiv 25 Apr 2023.

## Builds on (most cited first, 11 of 11)

- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [An Analysis of the Automatic Bug Fixing Performance of ChatGPT](https://mlsanity.com/paper/2301.08653/) (2023)
- [Exploring the Feasibility of ChatGPT for Event Extraction](https://mlsanity.com/paper/2303.03836/) (2023)
- [NLP-based Automated Compliance Checking of Data Processing Agreements against GDPR](https://mlsanity.com/paper/2209.09722/) (2022)
- [Generating Question Titles for Stack Overflow from Mined Code Snippets](https://mlsanity.com/paper/2005.10157/) (2020)
- [Automatically Matching Bug Reports With Related App Reviews](https://mlsanity.com/paper/2102.07134/) (2021)
- [Machine Learning for Software Engineering: A Tertiary Study](https://mlsanity.com/paper/2211.09425/) (2022)
- [Strategies, Benefits and Challenges of App Store-inspired Requirements Elicitation](https://mlsanity.com/paper/2301.12090/) (2023)
- [Emerging App Issue Identification via Online Joint Sentiment-Topic Tracing](https://mlsanity.com/paper/2008.09976/) (2020)

## Built on by (most cited first, 6 of 6)

- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future](https://mlsanity.com/paper/2408.02479/) (2024)
- [Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs](https://mlsanity.com/paper/2410.14641/) (2024)
- [Lessons from the Use of Natural Language Inference (NLI) in Requirements Engineering Tasks](https://mlsanity.com/paper/2405.05135/) (2024)
- [ERank: Fusing Supervised Fine-Tuning and Reinforcement Learning for Effective and Efficient Text Reranking](https://mlsanity.com/paper/2509.00520/) (2025)
- [Towards Communication-Efficient Adversarial Federated Learning for Robust Edge Intelligence](https://mlsanity.com/paper/2501.15257/) (2025)

## Abstract (arXiv)

Recently, various illustrative examples have shown the impressive ability of generative large language models (LLMs) to perform NLP related tasks. ChatGPT undoubtedly is the most representative model. We empirically evaluate ChatGPT's performance on requirements information retrieval (IR) tasks to derive insights into designing or developing more effective requirements retrieval methods or tools based on generative LLMs. We design an evaluation framework considering four different combinations of two popular IR tasks and two common artifact types. Under zero-shot setting, evaluation results reveal ChatGPT's promising ability to retrieve requirements relevant information (high recall) and limited ability to retrieve more specific requirements information (low precision). Our evaluation of ChatGPT on requirements IR under zero-shot setting provides preliminary evidence for designing or developing more effective requirements IR methods or tools based on LLMs.