# How good are LLMs at Retrieving Documents in a Specific Domain?

arXiv: https://arxiv.org/abs/2509.22658 · HTML: https://mlsanity.com/paper/2509.22658/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 25 Aug 2025.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment](https://mlsanity.com/paper/2312.12148/) (2023)
- [G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering](https://mlsanity.com/paper/2402.07630/) (2024)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunities](https://mlsanity.com/paper/2307.16376/) (2023)
- [STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases](https://mlsanity.com/paper/2404.13207/) (2024)
- [Interactive-KBQA: Multi-Turn Interactions for Knowledge Base Question Answering with Large Language Models](https://mlsanity.com/paper/2402.15131/) (2024)
- [Towards a Search Engine for Machines: Unified Ranking for Multiple Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.00175/) (2024)
- [Large Search Model: Redefining Search Stack in the Era of LLMs](https://mlsanity.com/paper/2310.14587/) (2023)
- [The Use of Generative Search Engines for Knowledge Work and Complex Tasks](https://mlsanity.com/paper/2404.04268/) (2024)
- [A Survey of Generative Search and Recommendation in the Era of Large Language Models](https://mlsanity.com/paper/2404.16924/) (2024)
- [Evaluating the Retrieval Component in LLM-Based Question Answering Systems](https://mlsanity.com/paper/2406.06458/) (2024)
- [A Comparative Study on TF-IDF feature Weighting Method and its Analysis using Unstructured Dataset](https://mlsanity.com/paper/2308.04037/) (2023)

## Abstract (arXiv)

Classical search engines using indexing methods in data infrastructures primarily allow keyword-based queries to retrieve content. While these indexing-based methods are highly scalable and efficient, due to a lack of an appropriate evaluation dataset and a limited understanding of semantics, they often fail to capture the user's intent and generate incomplete responses during evaluation. This problem also extends to domain-specific search systems that utilize a Knowledge Base (KB) to access data from various research infrastructures. Research infrastructures (RIs) from the environmental and earth science domain, which encompass the study of ecosystems, biodiversity, oceanography, and climate change, generate, share, and reuse large volumes of data. While there are attempts to provide a centralized search service using Elasticsearch as a knowledge base, they also face similar challenges in understanding queries with multiple intents. To address these challenges, we proposed an automated method to curate a domain-specific evaluation dataset to analyze the capability of a search system. Furthermore, we incorporate the Retrieval of Augmented Generation (RAG), powered by Large Language Models (LLMs), for high-quality retrieval of environmental domain data using natural language queries. Our quantitative and qualitative analysis of the evaluation dataset shows that LLM-based systems for information retrieval return results with higher precision when understanding queries with multiple intents, compared to Elasticsearch-based systems.