mlsanityData to 14 Oct 2026

TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain

arXiv 2412.15891 ↗First posted 20 Dec 2024cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Despite outstanding processes in many tasks, Large Language Models (LLMs) still lack accuracy when dealing with highly technical domains. Especially, telecommunications (telco) is a particularly challenging domain due the large amount of lexical, semantic and conceptual peculiarities. Yet, this domain holds many valuable use cases, directly linked to industrial needs. Hence, this paper studies how LLMs can be adapted to the telco domain. It reports our effort to (i) collect a massive corpus of domain-specific data (800M tokens, 80K instructions), (ii) perform adaptation using various methodologies, and (iii) benchmark them against larger generalist models in downstream tasks that require extensive knowledge of telecommunications. Our experiments on Llama-2-7b show that domain-adapted models can challenge the large generalist models. They also suggest that adaptation can be restricted to a unique instruction-tuning step, dicarding the need for any fine-tuning on raw texts beforehand.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 38 of its 47 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

38 of 47 references.

References · 38

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2023Mistral 7Bcited by 3,164
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
26 more
  1. 2024Mixtral of Expertscited by 1,823
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  5. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  6. 2023Textbooks Are All You Needcited by 569
  7. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  8. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  9. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  10. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  11. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  12. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  13. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 245
  14. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  15. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
  16. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 174
  17. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 173
  18. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  19. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  20. 2023TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledgecited by 63
  21. 2023Large Generative AI Models for Telecom: The Next Big Thing?cited by 52
  22. 2023Adapting Large Language Models to Domains via Reading Comprehensioncited by 50
  23. 2023Large Language Models for Telecom: Forthcoming Impact on the Industrycited by 44
  24. 2023Understanding Telecom Language Through Large Language Modelscited by 32
  25. 2023Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Datacited by 19
  26. 2021Fine-Tuning Transformers: Vocabulary Transfercited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Beyond Benchmarks: LLM Evaluation with an Anthropomorphic and Lifecycle-oriented Roadmapcited by 1
  2. 2026TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications?cited by 1
  3. 2026TelecomGPT-R1: Unified Post-Training for Reasoning Across Heterogeneous Telecom Taskscited by 1