mlsanityData to 7 Oct 2026

Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answering

arXiv 2305.11541 ↗First posted 19 May 2023cs.CL cs.AIcited by 41

Not on the map. Why?

Abstract (arXiv)

Large Language Model (LLM) has gained popularity and achieved remarkable results in open-domain tasks, but its performance in real industrial domain-specific scenarios is average due to its lack of specific domain knowledge. This issue has attracted widespread attention, but there are few relevant benchmarks available. In this paper, we provide a benchmark Question Answering (QA) dataset named MSQA, centered around Microsoft products and IT technical problems encountered by customers. This dataset contains industry cloud-specific QA knowledge, an area not extensively covered in general LLMs, making it well-suited for evaluating methods aiming to enhance LLMs' domain-specific capabilities. In addition, we propose a new model interaction paradigm that can empower LLM to achieve better performance on domain-specific tasks where it is not proficient. Extensive experiments demonstrate that the approach following our method outperforms the commonly used LLM with retrieval methods. We make our source code and sample data available at: https://aka.ms/Microsoft_QA.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

35 of 57 references.

References · 35

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
23 more
  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  5. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  6. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  8. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  9. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  10. 2023Instruction Tuning with GPT-4cited by 676
  11. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  12. 2019ELI5: Long Form Question Answeringcited by 567
  13. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  14. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  15. 2023Augmented Language Models: a Surveycited by 398
  16. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  17. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  18. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  19. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  20. 2015The Goldilocks Principle: Reading Children's Books with Explicit Memory Representationscited by 108
  21. 2019TWEETQA: A Social Media Focused Question Answering Datasetcited by 55
  22. 2022Read before Generate! Faithful Long Form Question Answering with Machine Readingcited by 47
  23. 2022Modern Question Answering Datasets and Benchmarks: A Surveycited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 20; 2025: 10; 2026: 5.

41 papers.

Sort:

By area:

  1. 2024BioRAG: A RAG-LLM Framework for Biological Question Reasoningcited by 42
  2. 2023Evaluating Open-QA Evaluationcited by 35
  3. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33
  4. 2023Large Language Models are Complex Table Parserscited by 28
  5. 2024Exploring the Impact of Table-to-Text Methods on Augmenting LLM-based Question Answering with Domain Hybrid Datacited by 24
  6. 2024T-RAG: Lessons from the LLM Trenchescited by 23
  7. 2024LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Surveycited by 12
  8. 2024KG-Retriever: Efficient Knowledge Indexing for Retrieval-Augmented Large Language Modelscited by 12
  9. 2025IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarioscited by 12
  10. 2024Nissist: An Incident Mitigation Copilot based on Troubleshooting Guidescited by 11
  11. 2024Towards Optimizing and Evaluating a Retrieval Augmented QA Chatbot using LLMs with Human in the Loopcited by 10
  12. 2023A Unified Industrial Large Knowledge Model Framework in Industry 4.0 and Smart Manufacturingcited by 8
29 more
  1. 2024Token-level Proximal Policy Optimization for Query Generationcited by 7
  2. 2025QuIM-RAG: Advancing Retrieval-Augmented Generation with Inverted Question Matching for Enhanced QA Performancecited by 6
  3. 2023Self-Specialization: Uncovering Latent Expertise within Large Language Modelscited by 5
  4. 2025ArgInstruct: Specialized Instruction Fine-Tuning for Computational Argumentationcited by 4
  5. 2025BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactionscited by 4
  6. 2024Engineering Conversational Search Systems: A Review of Applications, Architectures, and Functional Componentscited by 3
  7. 2024No Size Fits All: The Perils and Pitfalls of Leveraging LLMs Vary with Company Sizecited by 3
  8. 2024Quantifying Qualitative Insights: Leveraging LLMs to Market Predictcited by 3
  9. 2025ComRAG: Retrieval-Augmented Generation with Dynamic Vector Stores for Real-time Community Question Answering in Industrycited by 3
  10. 2025Select to Know: An Internal-External Knowledge Self-Selection Framework for Domain-Specific Question Answeringcited by 3
  11. 2023PACE-LM: Prompting and Augmentation for Calibrated Confidence Estimation with GPT-4 in Cloud Incident Root Cause Analysiscited by 2
  12. 2023Machine Learning Approaches for Diagnostics and Prognostics of Industrial Systems Using Open Source Data from PHM Data Challenges: A Reviewcited by 2
  13. 2024Knowledge Sharing in Manufacturing using Large Language Models: User Evaluation and Model Benchmarkingcited by 2
  14. 2024KaPQA: Knowledge-Augmented Product Question-Answeringcited by 2
  15. 2024Evaluating ChatGPT on Nuclear Domain-Specific Datacited by 2
  16. 2024AI Delegates with a Dual Focus: Ensuring Privacy and Strategic Self-Disclosurecited by 2
  17. 2025Fine-Tuning Large Language Models and Evaluating Retrieval Methods for Improved Question Answering on Building Codescited by 2
  18. 2025Atomic Consistency Preference Optimization for Long-Form Question Answeringcited by 1
  19. 2024ColBERT Retrieval and Ensemble Response Scoring for Language Model Question Answeringcited by 0
  20. 2024Task-oriented Prompt Enhancement via Script Generationcited by 0
  21. 2024Towards Optimizing a Retrieval Augmented Generation using Large Language Model on Academic Datacited by 0
  22. 2024An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problemscited by 0
  23. 2025Rethinking Data: Towards Better Performing Domain-Specific Small Language Modelscited by 0
  24. 2025A Collaborative Framework Integrating Large Language Model and Chemical Fragment Space: Mutual Inspiration for Lead Designcited by 0
  25. 2026MiCU: End-to-End Smart Home Command Understanding with Large Language Modelcited by 0
  26. 2026Evaluating LLMs' Effectiveness on Real-World Consumer Device Repair Questionscited by 0
  27. 2026AdaptAgent: A Multi-agent, Domain-Guided Reasoning Framework for Code Adaptationcited by 0
  28. 2026API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfacescited by 0
  29. 2026Domain-Specific Jargon in Large Language Models: A Comparative Analysis between General-Purpose and Specialist Modelscited by 0