# Telco-DPR: A Hybrid Dataset for Evaluating Retrieval Models of 3GPP Technical Specifications

arXiv: https://arxiv.org/abs/2410.19790 · HTML: https://mlsanity.com/paper/2410.19790/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 21 of 21)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [Efficient Natural Language Response Suggestion for Smart Reply](https://mlsanity.com/paper/1705.00652/) (2017)
- [Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering](https://mlsanity.com/paper/2101.00774/) (2021)
- [Open Question Answering over Tables and Text](https://mlsanity.com/paper/2010.10439/) (2020)
- [Simple Entity-Centric Questions Challenge Dense Retrievers](https://mlsanity.com/paper/2109.08535/) (2021)
- [Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities](https://mlsanity.com/paper/2405.10825/) (2024)
- [TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge](https://mlsanity.com/paper/2310.15051/) (2023)
- [Large Language Models for Telecom: Forthcoming Impact on the Industry](https://mlsanity.com/paper/2308.06013/) (2023)
- [A Survey on Machine Reading Comprehension Systems](https://mlsanity.com/paper/2001.01582/) (2020)
- [Dense Hierarchical Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2110.15439/) (2021)
- [TSpec-LLM: An Open-source Dataset for LLM Understanding of 3GPP Specifications](https://mlsanity.com/paper/2406.01768/) (2024)
- [Telco-RAG: Navigating the Challenges of Retrieval-Augmented Language Models for Telecommunications](https://mlsanity.com/paper/2404.15939/) (2024)
- [Exploring the Impact of Table-to-Text Methods on Augmenting LLM-based Question Answering with Domain Hybrid Data](https://mlsanity.com/paper/2402.12869/) (2024)
- [SPEC5G: A Dataset for 5G Cellular Network Protocol Analysis](https://mlsanity.com/paper/2301.09201/) (2023)
- [AI-native Interconnect Framework for Integration of Large Language Model Technologies in 6G Systems](https://mlsanity.com/paper/2311.05842/) (2023)
- [TelecomRAG: Taming Telecom Standards with Retrieval Augmented Generation and LLMs](https://mlsanity.com/paper/2406.07053/) (2024)
- [Telecom Language Models: Must They Be Large?](https://mlsanity.com/paper/2403.04666/) (2024)
- [Lecture Notes on Neural Information Retrieval](https://mlsanity.com/paper/2207.13443/) (2022)

## Built on by (most cited first, 3 of 3)

- [A Graph-based Approach for Multi-Modal Question Answering from Flowcharts in Telecom Documents](https://mlsanity.com/paper/2507.22938/) (2025)
- [Retrieval-Augmented Generation for Reliable Interpretation of Radio Regulations](https://mlsanity.com/paper/2509.09651/) (2025)
- [TelcoAI: Advancing 3GPP Technical Specification Search through Agentic Multi-Modal Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.16984/) (2025)

## Abstract (arXiv)

This paper proposes a Question-Answering (QA) system for the telecom domain using 3rd Generation Partnership Project (3GPP) technical documents. Alongside, a hybrid dataset, Telco-DPR, which consists of a curated 3GPP corpus in a hybrid format, combining text and tables, is presented. Additionally, the dataset includes a set of synthetic question/answer pairs designed to evaluate the retrieval performance of QA systems on this type of data. The retrieval models, including the sparse model, Best Matching 25 (BM25), as well as dense models, such as Dense Passage Retriever (DPR) and Dense Hierarchical Retrieval (DHR), are evaluated and compared using top-K accuracy and Mean Reciprocal Rank (MRR). The results show that DHR, a retriever model utilising hierarchical passage selection through fine-tuning at both the document and passage levels, outperforms traditional methods in retrieving relevant technical information, achieving a Top-10 accuracy of 86.2%. Additionally, the Retriever-Augmented Generation (RAG) technique, used in the proposed QA system, is evaluated to demonstrate the benefits of using the hybrid dataset and the DHR. The proposed QA system, using the developed RAG model and the Generative Pretrained Transformer (GPT)-4, achieves a 14% improvement in answer accuracy, when compared to a previous benchmark on the same dataset.