mlsanityData to 7 Oct 2026

Online Cascade Learning for Efficient Inference over Streams

arXiv 2402.04513 ↗First posted 7 Feb 2024cs.LG cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have a natural role in answering complex queries about data streams, but the high computational cost of LLM inference makes them infeasible in many such tasks. We propose online cascade learning, the first approach to address this challenge. The objective here is to learn a "cascade" of models, starting with lower-capacity models (such as logistic regression) and ending with a powerful LLM, along with a deferral policy that determines the model to be used on a given input. We formulate the task of learning cascades online as an imitation-learning problem, where smaller models are updated over time imitating the collected LLM demonstrations, and give a no-regret algorithm for the problem. Experimental results across four benchmarks show that our method parallels LLMs in accuracy while cutting down inference costs by as much as 90% with strong robustness against input distribution shifts, underscoring its efficacy and adaptability in stream processing.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

28 of 36 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  9. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  10. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  11. 2019On the Efficacy of Knowledge Distillationcited by 377
  12. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
16 more
  1. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  2. 2018Online Learning: A Comprehensive Surveycited by 203
  3. 2018Hate Speech Dataset from a White Supremacy Forumcited by 196
  4. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
  5. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  6. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  7. 2023Online Speculative Decodingcited by 109
  8. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 58
  9. 2020CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascadecited by 47
  10. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 43
  11. 2023Small Models are Valuable Plug-ins for Large Language Modelscited by 42
  12. 2020Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Modelscited by 39
  13. 2022BabyBear: Cheap inference triage for expensive language modelscited by 10
  14. 2021When in Doubt, Summon the Titans: Efficient Inference with Large Modelscited by 8
  15. 2023Cache me if you Can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Modelscited by 4
  16. 2023Cache & Distil: Optimising API Calls to Large Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 9; 2026: 6.

19 papers.

Sort:

By area:

  1. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  2. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  3. 2024A Unified Approach to Routing and Cascading for LLMscited by 71
  4. 2024Agreement-Based Cascading for Efficient Inferencecited by 16
  5. 2025RadialRouter: Structured Representation for Efficient and Robust Large Language Models Routingcited by 11
  6. 2025Gatekeeper: Improving Model Cascades Through Confidence Tuningcited by 10
  7. 2025Toward Super Agent System with Hybrid AI Routerscited by 6
  8. 2025Resource-efficient Inference with Foundation Model Programscited by 2
  9. 2025Bi-directional Model Cascading with Proxy Confidencecited by 2
  10. 2026Streaming Model Cascades for Semantic SQLcited by 2
  11. 2024MODL: Multilearner Online Deep Learningcited by 1
  12. 2025From Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascadescited by 1
7 more
  1. 2025Detect, Explain, Escalate: Sustainable Dialogue Breakdown Management for LLM Agentscited by 0
  2. 2025T-TAMER: Provably Taming Trade-offs in ML Servingcited by 0
  3. 2026Bridging Local and Global Knowledge: Cascaded Mixture-of-Experts Learning for Near-Shortest Path Routingcited by 0
  4. 2026When Efficiency Backfires: Cascading LLMs Trigger Cascade Failure under Adversarial Attackcited by 0
  5. 2026SWRouter: Similarity-Contractive Window Routing for Multi-Turn Large Language Model Conversationscited by 0
  6. 2026When to Call an LLM: A Confidence-Gated Hybrid for Cost-Effective Emotion Recognition in Conversational AIcited by 0
  7. 2026MAADBench: The Refreshable Paradigm for Anomaly Detection in Multi-Agent Systemscited by 0