# TelecomGPT: A Framework to Build Telecom-Specfic Large Language Models

arXiv: https://arxiv.org/abs/2407.09424 · HTML: https://mlsanity.com/paper/2407.09424/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 76 references that are in the map.
- First posted on arXiv 12 Jul 2024.

## Builds on (most cited first, 30 of 64)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [Llemma: An Open Language Model For Mathematics](https://mlsanity.com/paper/2310.10631/) (2023)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)

## Built on by (most cited first, 30 of 40)

- [Tele-LLMs: A Series of Specialized Large Language Models for Telecommunications](https://mlsanity.com/paper/2409.05314/) (2024)
- [A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directions](https://mlsanity.com/paper/2412.19823/) (2024)
- [Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions](https://mlsanity.com/paper/2501.04437/) (2025)
- [Diffusion Models as Network Optimizers: Explorations and Analysis](https://mlsanity.com/paper/2411.00453/) (2024)
- [ORANSight-2.0: Foundational LLMs for O-RAN](https://mlsanity.com/paper/2503.05200/) (2025)
- [Agentic AI Empowered Intent-Based Networking for 6G](https://mlsanity.com/paper/2601.06640/) (2026)
- [WirelessAgent++: Automated Agentic Workflow Design and Benchmarking for Wireless Networks](https://mlsanity.com/paper/2603.00501/) (2026)
- [RF-GPT: Teaching AI to See the Wireless World](https://mlsanity.com/paper/2602.14833/) (2026)
- [Chat3GPP: An Open-Source Retrieval-Augmented Generation Framework for 3GPP Documents](https://mlsanity.com/paper/2501.13954/) (2025)
- [TeleMath: A Benchmark for Large Language Models in Telecom Mathematical Problem Solving](https://mlsanity.com/paper/2506.10674/) (2025)
- [Large Language Model as a Catalyst: A Paradigm Shift in Base Station Siting Optimization](https://mlsanity.com/paper/2408.03631/) (2024)
- [NetOrchLLM: Mastering Wireless Network Orchestration with Large Language Models](https://mlsanity.com/paper/2412.10107/) (2024)
- [Telco-oRAG: Optimizing Retrieval-augmented Generation for Telecom Queries via Hybrid Retrieval and Neural Routing](https://mlsanity.com/paper/2505.11856/) (2025)
- [From Large AI Models to Agentic AI: A Tutorial on Future Intelligent Communications](https://mlsanity.com/paper/2505.22311/) (2025)
- [Benchmarking Vector, Graph and Hybrid Retrieval Augmented Generation (RAG) Pipelines for Open Radio Access Networks (ORAN)](https://mlsanity.com/paper/2507.03608/) (2025)
- [MobiLLM: An Agentic AI Framework for Closed-Loop Threat Mitigation in 6G Open RANs](https://mlsanity.com/paper/2509.21634/) (2025)
- [Large Model Enabled Embodied Intelligence for 6G Integrated Perception, Communication, and Computation Network](https://mlsanity.com/paper/2512.15109/) (2025)
- [Beyond Benchmarks: LLM Evaluation with an Anthropomorphic and Lifecycle-oriented Roadmap](https://mlsanity.com/paper/2508.18646/) (2025)
- [Retrieval-Augmented Generation for Reliable Interpretation of Radio Regulations](https://mlsanity.com/paper/2509.09651/) (2025)
- [Think Less, Label Better: Multi-Stage Domain-Grounded Synthetic Data Generation for Fine-Tuning Large Language Models in Telecommunications](https://mlsanity.com/paper/2509.25736/) (2025)
- [TeleDoCTR: Domain-Specific and Contextual Troubleshooting for Telecommunications](https://mlsanity.com/paper/2601.00691/) (2026)
- [BeamAgent: LLM-Aided MIMO Beamforming with Decoupled Intent Parsing and Alternating Optimization for Joint Site Selection and Precoding](https://mlsanity.com/paper/2603.18855/) (2026)
- [Agentic AI-Based Joint Computing and Networking via Mixture of Experts and Large Language Models](https://mlsanity.com/paper/2605.02911/) (2026)
- [TelecomGPT-R1: Unified Post-Training for Reasoning Across Heterogeneous Telecom Tasks](https://mlsanity.com/paper/2609.25356/) (2026)
- [System One Models for Wireless Decision-Making:Applications and Performance Evaluation](https://mlsanity.com/paper/2610.04345/) (2026)
- [Empowering Large Language Models in Wireless Communication: A Novel Dataset and Fine-Tuning Framework](https://mlsanity.com/paper/2501.09631/) (2025)
- [Knowledge Distillation of Domain-adapted LLMs for Question-Answering in Telecom](https://mlsanity.com/paper/2504.20000/) (2025)
- [MM-Telco: Benchmarks and Multimodal Large Language Models for Telecom Applications](https://mlsanity.com/paper/2511.13131/) (2025)
- [Customized User Plane Processing via Code Generating AI Agents for Next Generation Mobile Networks](https://mlsanity.com/paper/2604.03282/) (2026)
- [When Does Multimodal AI Help? Diagnostic Complementarity of Vision-Language Models and CNNs for Spectrum Management in Satellite-Terrestrial Networks](https://mlsanity.com/paper/2604.03774/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have the potential to revolutionize the Sixth Generation (6G) communication networks. However, current mainstream LLMs generally lack the specialized knowledge in telecom domain. In this paper, for the first time, we propose a pipeline to adapt any general purpose LLMs to a telecom-specific LLMs. We collect and build telecom-specific pre-train dataset, instruction dataset, preference dataset to perform continual pre-training, instruct tuning and alignment tuning respectively. Besides, due to the lack of widely accepted evaluation benchmarks in telecom domain, we extend existing evaluation benchmarks and proposed three new benchmarks, namely, Telecom Math Modeling, Telecom Open QnA and Telecom Code Tasks. These new benchmarks provide a holistic evaluation of the capabilities of LLMs including math modeling, Open-Ended question answering, code generation, infilling, summarization and analysis in telecom domain. Our fine-tuned LLM TelecomGPT outperforms state of the art (SOTA) LLMs including GPT-4, Llama-3 and Mistral in Telecom Math Modeling benchmark significantly and achieve comparable performance in various evaluation benchmarks such as TeleQnA, 3GPP technical documents classification, telecom code summary and generation and infilling.