mlsanityData to 7 Oct 2026

Language Models as a Service: Overview of a New Paradigm and its Challenges

arXiv 2309.16573 ↗First posted 28 Sep 2023cs.AI cs.CL cs.CYcited by 10

Not on the map. Why?

Abstract (arXiv)

Some of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the Language-Models-as-a-Service (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case of open-source models, such closed-off language models present specific challenges for evaluating, benchmarking, and testing them. This paper has two goals: on the one hand, we delineate how the aforementioned challenges act as impediments to the accessibility, replicability, reliability, and trustworthiness of LMaaS. We systematically examine the issues that arise from a lack of information about language models for each of these four aspects. We conduct a detailed analysis of existing solutions and put forth a number of considered recommendations, and highlight the directions for future advancements. On the other hand, it serves as a comprehensive resource for existing knowledge on current, major LMaaS, offering a synthesized overview of the licences and capabilities their interfaces offer.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 98 of its 141 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

98 of 141 references.

References · 98

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2020Extracting Training Data from Large Language Modelscited by 2,144
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  13. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  14. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  15. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  16. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  17. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  18. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  19. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  20. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  21. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  22. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  23. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  24. 2023Whose Opinions Do Language Models Reflect?cited by 759
  25. 2022A Survey on In-context Learningcited by 717
  26. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  27. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  28. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  29. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  30. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 536
  31. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  32. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  33. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  34. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  35. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  36. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  37. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  38. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399

50 most cited of 98 All 98 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 2.

10 papers.

  1. 2024A Scalable Communication Protocol for Networks of Large Language Modelscited by 43
  2. 2025Large Language Models Miss the Multi-Agent Markcited by 25
  3. 2025Is Your LLM Overcharging You? Tokenization, Transparency, and Incentivescited by 13
  4. 2024Data for Mathematical Copilots: Better Ways of Presenting Proofs for Machine Learningcited by 11
  5. 2024An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factorscited by 6
  6. 2025Auditing Pay-Per-Token in Large Language Modelscited by 4
  7. 2025ML-Asset Management: Curation, Discovery, and Utilizationcited by 1
  8. 2025Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-servicecited by 0
  9. 2026The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Playcited by 0
  10. 2026NL-PAC: Specification Ambiguity and Certified Minimax Risk Floors in LLM-Mediated Supervisioncited by 0