mlsanityData to 7 Oct 2026

Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving

arXiv 2312.05385 ↗First posted 8 Dec 2023cs.DC cs.LGcited by 15

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Machine learning (ML) inference platforms are tasked with balancing two competing goals: ensuring high throughput given many requests, and delivering low-latency responses to support interactive applications. Unfortunately, existing platform knobs (e.g., batch sizes) fail to ease this fundamental tension, and instead only enable users to harshly trade off one property for the other. This paper explores an alternate strategy to taming throughput-latency tradeoffs by changing the granularity at which inference is performed. We present Apparate, a system that automatically applies and manages early exits (EEs) in ML models, whereby certain inputs can exit with results at intermediate layers. To cope with the time-varying overhead and accuracy challenges that EEs bring, Apparate repurposes exits to provide continual feedback that powers several novel runtime monitoring and adaptation strategies. Apparate lowers median response latencies by 40.5--91.5% and 10.0--24.2% for diverse CV and NLP classification workloads, and median time-per-token latencies by 22.6--77.9% for generative scenarios, without affecting throughputs or violating tight accuracy constraints.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 35 of its 85 references that are in the map. First posted on arXiv 8 Dec 2023.

Builds on

35 of 85 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Confident Adaptive Language Modeling14 Jul
  2. 2021Elbert: Fast Albert with Confidence-Window Based Early Exit1 Jul

Other references · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  10. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  12. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
21 more
  1. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  2. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  3. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  4. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  5. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  6. 2017Modulating early visual processing by languagecited by 167
  7. 2017Adaptive Neural Networks for Efficient Inferencecited by 161
  8. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  9. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  10. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  11. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  12. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
  13. 2020Ekya: Continuous Learning of Video Analytics Models on Edge Compute Serverscited by 58
  14. 2018Dynamic Video Segmentation Networkcited by 51
  15. 2018Focus: Querying Large Video Datasets with Low Latency and Low Costcited by 46
  16. 2019Scaling Video Analytics on Constrained Edge Nodescited by 36
  17. 2018Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implicationscited by 35
  18. 2022GEMEL: Model Merging for Memory-Efficient, Real-Time Video Analytics at the Edgecited by 32
  19. 2020LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inferencecited by 21
  20. 2019Preech: A System for Privacy-Preserving Speech Transcriptioncited by 13
  21. 2021Boggart: Towards General-Purpose Acceleration of Retrospective Video Analyticscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 4.

15 papers.

Sort:

By area:

  1. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 17
  2. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16
  3. 2024EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Modelscited by 13
  4. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Servingcited by 7
  5. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Servingcited by 5
  6. 2025SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Servingcited by 5
  7. 2025FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Frameworkcited by 3
  8. 2026Agentic Witnessing: Pragmatic and Scalable TEE-Enabled Privacy-Preserving Auditingcited by 3
  9. 2025Dynamic Rebatching for Efficient Early-Exit Inference with DREXcited by 2
  10. 2026DiLaServe: High SLO Attainment Serving for Diffusion Language Modelscited by 1
  11. 2025Legilimens: Performant Video Analytics on the System-on-Chip Edgecited by 0
  12. 2025T-TAMER: Provably Taming Trade-offs in ML Servingcited by 0
3 more
  1. 2025RedunCut: Measurement-Driven Sampling and Accuracy Performance Modeling for Low-Cost Live Video Analyticscited by 0
  2. 2026EdgeServing: Deadline-Aware Multi-DNN Serving at the Edgecited by 0
  3. 2026vSkipper: Translating Dynamic Layer Skipping into LLM Serving Gainscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inference
  2. 2024RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inference
  3. 2023Batch Prompting: Efficient Inference with Large Language Model APIs
  4. 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving
  5. 2024An Efficient Inference Framework for Early-exit Large Language Models
  6. 2024Performance Control in Early Exiting to Deploy Large Models at the Same Cost of Smaller Ones
  7. 2023Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference in Low Resource Settings
  8. 2026Test-Time Augmentation for LLMs: When Input Diversity Beats Output Diversity at Matched Compute