Towards a Middleware for Large Language Models
Not on the map. Why?
Abstract (arXiv)
Large language models have gained widespread popularity for their ability to process natural language inputs and generate insights derived from their training data, nearing the qualities of true artificial intelligence. This advancement has prompted enterprises worldwide to integrate LLMs into their services. So far, this effort is dominated by commercial cloud-based solutions like OpenAI's ChatGPT and Microsoft Azure. As the technology matures, however, there is a strong incentive for independence from major cloud providers through self-hosting "LLM as a Service", driven by privacy, cost, and customization needs. In practice, hosting LLMs independently presents significant challenges due to their complexity and integration issues with existing systems. In this paper, we discuss our vision for a forward-looking middleware system architecture that facilitates the deployment and adoption of LLMs in enterprises, even for advanced use cases in which we foresee LLMs to serve as gateways to a complete application ecosystem and, to some degree, absorb functionality traditionally attributed to the middleware.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 21 of its 61 references that are in the map. First posted on arXiv 21 Nov 2024.
Builds on
21 of 61 references.
References · 21
- 2017Attention Is All You Needcited by 59,932
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
- 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
- 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
- 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
- 2022Efficiently Scaling Transformer Inferencecited by 529
- 2020Learning under Concept Drift: A Reviewcited by 470
- 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
- 2023Large Language Models for Information Retrieval: A Surveycited by 331
- 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
9 more
- 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
- 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
- 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
- 2021Dialogue State Tracking with a Language Model using Schema-Driven Promptingcited by 94
- 2023A Survey on Out-of-Distribution Detection in NLPcited by 31
- 2023Large Language Models are Built-in Autoregressive Search Enginescited by 26
- 2017Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practicescited by 22
- 2024NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attentioncited by 12
- 2020A Qualitative Study of Application-level Cachingcited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.