mlsanityData to 14 Oct 2026

Scaling Down to Scale Up: A Cost-Benefit Analysis of Replacing OpenAI's LLM with Open Source SLMs in Production

arXiv 2312.14972 ↗First posted 20 Dec 2023cs.SE cs.AI cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Many companies use large language models (LLMs) offered as a service, like OpenAI's GPT-4, to create AI-enabled product experiences. Along with the benefits of ease-of-use and shortened time-to-solution, this reliance on proprietary services has downsides in model control, performance reliability, uptime predictability, and cost. At the same time, a flurry of open-source small language models (SLMs) has been made available for commercial use. However, their readiness to replace existing capabilities remains unclear, and a systematic approach to holistically evaluate these SLMs is not readily available. This paper presents a systematic evaluation methodology and a characterization of modern open-source SLMs and their trade-offs when replacing proprietary LLMs for a real-world product feature. We have designed SLaM, an open-source automated analysis tool that enables the quantitative and qualitative testing of product features utilizing arbitrary SLMs. Using SLaM, we examine the quality and performance characteristics of modern SLMs relative to an existing customer-facing implementation using the OpenAI GPT-4 API. Across 9 SLMs and their 29 variants, we observe that SLMs provide competitive results, significant performance consistency improvements, and a cost reduction of 5x~29x when compared to GPT-4.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 29 of its 61 references that are in the map. First posted on arXiv 20 Dec 2023.

Builds on

29 of 61 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  9. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  10. 2018Universal Sentence Encodercited by 489
  11. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  12. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
17 more
  1. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  2. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  3. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  4. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 64
  5. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 50
  6. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 48
  7. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  8. 2023Small Models are Valuable Plug-ins for Large Language Modelscited by 42
  9. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 39
  10. 2023Efficient LLM Inference on CPUscited by 24
  11. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 15
  12. 2022XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaversecited by 14
  13. 2023A Hardware Evaluation Framework for Large Language Model Inferencecited by 14
  14. 2022Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputationcited by 12
  15. 2023VEGETA: Vertically-Integrated Extensions for Sparse/Dense GEMM Tile Acceleration on CPUscited by 9
  16. 2023Effective Quantization for Diffusion Models on CPUscited by 7
  17. 2021Self-Adaptive Reconfigurable Arrays (SARA): Using ML to Assist Scaling GEMM Accelerationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 13; 2026: 2.

21 papers.

Sort:

By area:

  1. 2024SLM-Mod: Small Language Models Surpass LLMs at Content Moderationcited by 27
  2. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 20
  3. 2025Metamorphic Testing of Large Language Models for Natural Language Processingcited by 12
  4. 2025Bridging the LLM Accessibility Divide? Performance, Fairness, and Cost of Closed versus Open LLMs for Automated Essay Scoringcited by 7
  5. 2025Small Language Models for Efficient Agentic Tool Calling: Outperforming Large Models with Targeted Fine-tuningcited by 7
  6. 2024Rethinking Scale: The Efficacy of Fine-Tuned Open-Source LLMs in Large-Scale Reproducible Social Science Researchcited by 6
  7. 2025Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generationcited by 6
  8. 2024Benchmarking LLMs in Political Content Text-Annotation: Proof-of-Concept with Toxicity and Incivility Datacited by 4
  9. 2024Aligning Model Evaluations with Human Preferences: Mitigating Token Count Bias in Language Model Assessmentscited by 3
  10. 2024TOBUGraph: Knowledge Graph-Based Retrieval for Enhanced LLM Performance Beyond RAGcited by 3
  11. 2024From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmapcited by 2
  12. 2025Beyond Release: Access Considerations for Generative AI Systemscited by 1
9 more
  1. 2025Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applicationscited by 1
  2. 2025Small Language Models for Phishing Website Detection: Cost, Performance, and Privacy Trade-Offscited by 1
  3. 2025SLMEval: Entropy-Based Calibration for Human-Aligned Evaluation of Large Language Modelscited by 0
  4. 2025Bridging the Digital Divide: Small Language Models as a Pathway for Physics and Photonics Education in Underdeveloped Regionscited by 0
  5. 2025On-Premise SLMs vs. Commercial LLMs: Prompt Engineering and Incident Classification in SOCs and CSIRTscited by 0
  6. 2025LOCUS: A System and Method for Low-Cost Customization for Universal Specializationcited by 0
  7. 2025TrueBrief: Faithful Summarization through Small Language Modelscited by 0
  8. 2026Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidancecited by 0
  9. 2026Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUscited by 0