# Leveraging Fine-Tuned Retrieval-Augmented Generation with Long-Context Support: For 3GPP Standards

arXiv: https://arxiv.org/abs/2408.11775 · HTML: https://mlsanity.com/paper/2408.11775/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 21 Aug 2024.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 20 of 20)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [Large Language Model Unlearning](https://mlsanity.com/paper/2310.10683/) (2023)
- [LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning](https://mlsanity.com/paper/2401.01325/) (2024)
- [TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge](https://mlsanity.com/paper/2310.15051/) (2023)
- [Large Language Models for Telecom: Forthcoming Impact on the Industry](https://mlsanity.com/paper/2308.06013/) (2023)
- [Interactive AI with Retrieval-Augmented Generation for Next Generation Networking](https://mlsanity.com/paper/2401.11391/) (2024)
- [Understanding Telecom Language Through Large Language Models](https://mlsanity.com/paper/2306.07933/) (2023)
- [TSpec-LLM: An Open-source Dataset for LLM Understanding of 3GPP Specifications](https://mlsanity.com/paper/2406.01768/) (2024)
- [Telco-RAG: Navigating the Challenges of Retrieval-Augmented Language Models for Telecommunications](https://mlsanity.com/paper/2404.15939/) (2024)
- [ORAN-Bench-13K: An Open Source Benchmark for Assessing LLMs in Open Radio Access Networks](https://mlsanity.com/paper/2407.06245/) (2024)
- [SPEC5G: A Dataset for 5G Cellular Network Protocol Analysis](https://mlsanity.com/paper/2301.09201/) (2023)
- [Using Large Language Models to Understand Telecom Standards](https://mlsanity.com/paper/2404.02929/) (2024)
- [Linguistic Intelligence in Large Language Models for Telecommunications](https://mlsanity.com/paper/2402.15818/) (2024)
- [Telecom Language Models: Must They Be Large?](https://mlsanity.com/paper/2403.04666/) (2024)
- [Large Language Model-Driven Curriculum Design for Mobile Networks](https://mlsanity.com/paper/2405.18039/) (2024)

## Built on by (most cited first, 8 of 8)

- [LLMs for Explainable AI: A Comprehensive Survey](https://mlsanity.com/paper/2504.00125/) (2025)
- [Retrieval Augmented Generation with Multi-Modal LLM Framework for Wireless Environments](https://mlsanity.com/paper/2503.07670/) (2025)
- [Mapping the Landscape of Generative AI in Network Monitoring and Management](https://mlsanity.com/paper/2502.08576/) (2025)
- [TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Network](https://mlsanity.com/paper/2411.02617/) (2024)
- [TelecomGPT-R1: Unified Post-Training for Reasoning Across Heterogeneous Telecom Tasks](https://mlsanity.com/paper/2609.25356/) (2026)
- [WirelessAgent: Large Language Model Agents for Intelligent Wireless Networks](https://mlsanity.com/paper/2409.07964/) (2024)
- [TelcoAI: Advancing 3GPP Technical Specification Search through Agentic Multi-Modal Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.16984/) (2025)
- [Customized Generative AI Agent for Transportation Engineering Practice: A Development and Continued Pre-training Guideline](https://mlsanity.com/paper/2606.29014/) (2026)

## Abstract (arXiv)

Recent studies show that large language models (LLMs) struggle with technical standards in telecommunications. We propose a fine-tuned retrieval-augmented generation (RAG) system based on the Phi-2 small language model (SLM) to serve as an oracle for communication networks. Our developed system leverages forward-looking semantic chunking to adaptively determine parsing breakpoints based on embedding similarity, enabling effective processing of diverse document formats. To handle the challenge of multiple similar contexts in technical standards, we employ a re-ranking algorithm to prioritize the most relevant retrieved chunks. Recognizing the limitations of Phi-2's small context window, we implement a recent technique, namely SelfExtend, to expand the context window during inference, which not only boosts the performance but also can accommodate a wider range of user queries and design requirements from customers to specialized technicians. For fine-tuning, we utilize the low-rank adaptation (LoRA) technique to enhance computational efficiency during training and enable effective fine-tuning on small datasets. Our comprehensive experiments demonstrate substantial improvements over existing question-answering approaches in the telecom domain, achieving performance that exceeds larger language models such as GPT-4 (which is about 880 times larger in size). This work presents a novel approach to leveraging SLMs for communication networks, offering a balance of efficiency and performance. This work can serve as a foundation towards agentic language models for networks.