mlsanityData to 7 Oct 2026

Evaluating the Efficacy of Open-Source LLMs in Enterprise-Specific RAG Systems: A Comparative Study of Performance and Scalability

arXiv 2406.11424 ↗First posted 17 Jun 2024cs.IR cs.CLcited by 15

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

This paper presents an analysis of open-source large language models (LLMs) and their application in Retrieval-Augmented Generation (RAG) tasks, specific for enterprise-specific data sets scraped from their websites. With the increasing reliance on LLMs in natural language processing, it is crucial to evaluate their performance, accessibility, and integration within specific organizational contexts. This study examines various open-source LLMs, explores their integration into RAG frameworks using enterprise-specific data, and assesses the performance of different open-source embeddings in enhancing the retrieval and generation process. Our findings indicate that open-source LLMs, combined with effective embedding techniques, can significantly improve the accuracy and efficiency of RAG systems, offering a viable alternative to proprietary solutions for enterprises.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 11 of its 17 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

11 of 17 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024COS-Mix: Cosine Similarity and Distance Fusion for Improved Information Retrieval2 Jun
  2. 2024Evaluation of Retrieval-Augmented Generation: A Survey13 May
  3. 2024A Fine-tuning Enhanced RAG System with Quantized Influence Measure as AI Judge26 Feb

Other references · 8

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  5. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  6. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  7. 2023A Cost Analysis of Generative Language Models and Influence Operationscited by 17
  8. 2023Keyword Augmented Retrieval: Novel framework for Information Retrieval integrated with speech interfacecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 1.

15 papers.

Sort:

By area:

  1. 2024ORAN-Bench-13K: An Open Source Benchmark for Assessing LLMs in Open Radio Access Networkscited by 22
  2. 2024Introducing a new hyper-parameter for RAG: Context Window Utilizationcited by 8
  3. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  4. 2025KnowsLM: A framework for evaluation of small language models for knowledge augmentation and humanised conversationscited by 6
  5. 2025AI5GTest: AI-Driven Specification-Aware Automated Testing and Validation of 5G O-RAN Componentscited by 3
  6. 2025Distributed Retrieval-Augmented Generationcited by 2
  7. 2025Prescriptive Agents based on RAG for Automated Maintenance (PARAM)cited by 2
  8. 2025Can we Evaluate RAGs with Synthetic Data?cited by 2
  9. 2025i-LAVA: Insights on Low Latency Voice-2-Voice Architecture for Agentscited by 2
  10. 2026GSM-SEM: Benchmark and Framework for Generating Semantically Variant Augmentationscited by 2
  11. 2024CultureVo: The Serious Game of Utilizing Gen AI for Enhancing Cultural Intelligencecited by 1
  12. 2025E-ARMOR: Edge case Assessment and Review of Multilingual Optical Character Recognitioncited by 1
3 more
  1. 2025ZoFia: Zero-Shot Fake News Detection with Entity-Guided Retrieval and Multi-LLM Interactioncited by 1
  2. 2025Two Is Better Than One: Rotations Scale LoRAscited by 0
  3. 2025Emission-GPT: A domain-specific language model agent for knowledge retrieval, emission inventory and data analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RAG Does Not Work for Enterprises
  2. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  3. 2025An Agile Method for Implementing Retrieval Augmented Generation Tools in Industrial SMEs
  4. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  5. 2025Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation
  6. 2024Deploying Large Language Models With Retrieval Augmented Generation
  7. 2024T-RAG: Lessons from the LLM Trenches
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Survey