mlsanityData to 7 Oct 2026

Cache & Distil: Optimising API Calls to Large Language Models

arXiv 2310.13561 ↗First posted 20 Oct 2023cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Large-scale deployment of generative AI tools often depends on costly API calls to a Large Language Model (LLM) to fulfil user queries. To curtail the frequency of these calls, one can employ a smaller language model -- a student -- which is continuously trained on the responses of the LLM. This student gradually gains proficiency in independently handling an increasing number of user requests, a process we term neural caching. The crucial element in neural caching is a policy that decides which requests should be processed by the student alone and which should be redirected to the LLM, subsequently aiding the student's learning. In this study, we focus on classification tasks, and we consider a range of classic active learning-based selection criteria as the policy. Our experiments suggest that Margin Sampling and Query by Committee bring consistent benefits across tasks and budgets.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 24 of its 55 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

24 of 55 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  6. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  7. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  8. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  9. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  10. 2019Green AIcited by 620
  11. 2020A Survey of Deep Active Learningcited by 547
  12. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 507
12 more
  1. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  2. 2020Evaluation of Text Generation: A Surveycited by 275
  3. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  4. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  5. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 94
  6. 2022A Comparative Survey of Deep Active Learningcited by 88
  7. 2020MixKD: Towards Efficient Distillation of Large-scale Language Modelscited by 65
  8. 2021Revisiting Uncertainty-based Query Strategies for Active Learning with Transformerscited by 59
  9. 2020Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixupcited by 22
  10. 2023On Optimal Caching and Model Multiplexing for Large Model Inferencecited by 20
  11. 2019Certainty Driven Consistency Loss on Multi-Teacher Networks for Semi-Supervised Learningcited by 11
  12. 2021ALLWAS: Active Learning on Language models in WASserstein spacecited by 1

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  2. 2024Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selectioncited by 32
  3. 2024Online Cascade Learning for Efficient Inference over Streamscited by 19
  4. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5