mlsanityData to 7 Oct 2026

QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling

arXiv 2409.14175 ↗First posted 21 Sep 2024cs.CL cs.AI cs.LGcited by 2

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Large Language models (LLMs) have brought about substantial advancements in the field of Question Answering (QA) systems. These models do remarkably well in addressing intricate inquiries in a variety of disciplines. However, because of domain-specific vocabulary, complex technological concepts, and the requirement for exact responses applying LLMs to specialized sectors like telecommunications presents additional obstacles. GPT-3.5 has been used in recent work, to obtain noteworthy accuracy for telecom-related questions in a Retrieval Augmented Generation (RAG) framework. Notwithstanding these developments, the practical use of models such as GPT-3.5 is restricted by their proprietary nature and high computing demands. This paper introduces QMOS, an innovative approach which uses a Question-Masked loss and Option Shuffling trick to enhance the performance of LLMs in answering Multiple-Choice Questions in the telecommunications domain. Our focus was on using opensource, smaller language models (Phi-2 and Falcon-7B) within an enhanced RAG framework. Our multi-faceted approach involves several enhancements to the whole LLM-RAG pipeline of finetuning, retrieval, prompt engineering and inference. Our approaches significantly outperform existing results, achieving accuracy improvements from baselines of 24.70% to 49.30% with Falcon-7B and from 42.07% to 84.65% with Phi-2.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 19 of its 27 references that are in the map. First posted on arXiv 21 Sep 2024.

Builds on

19 of 27 references.

References · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  7. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  8. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  9. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  10. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  11. 2023The Falcon Series of Open Language Modelscited by 526
  12. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
7 more
  1. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  2. 2023TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledgecited by 62
  3. 2024Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledgecited by 49
  4. 2024A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Taskscited by 46
  5. 2024Telco-RAG: Navigating the Challenges of Retrieval-Augmented Language Models for Telecommunicationscited by 26
  6. 2024Linguistic Intelligence in Large Language Models for Telecommunicationscited by 11
  7. 2024Telecom Language Models: Must They Be Large?cited by 10

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Networkcited by 2
  2. 2025Quantifying and Mitigating Selection Bias in LLMs: A Transferable LoRA Fine-Tuning and Efficient Majority Voting Approachcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025First Token Probability Guided RAG for Telecom Question Answering
  2. 2024Evaluating the Retrieval Component in LLM-Based Question Answering Systems
  3. 2026ARMOR: Adaptive Retriever Optimization for Low-Resource Telecom Question Answering
  4. 2025Enhancing Large Language Models (LLMs) for Telecommunications using Knowledge Graphs and Retrieval-Augmented Generation
  5. 2025Agri-Query: A Case Study on RAG vs. Long-Context LLMs for Cross-Lingual Technical Question Answering
  6. 2025Comparing the Performance of LLMs in RAG-based Question-Answering: A Case Study in Computer Science Literature
  7. 2024Evaluation of RAG Metrics for Question Answering in the Telecom Domain
  8. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity