mlsanityData to 7 Oct 2026

Inference acceleration for large language models using "stairs" assisted greedy generation

arXiv 2407.19947 ↗First posted 29 Jul 2024cs.CL cs.LGcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large Language Models (LLMs) with billions of parameters are known for their impressive predicting capabilities but require lots of resources to run. With their massive rise in popularity, even a small reduction in required resources could have an impact on environment. On the other hand, smaller models require fewer resources but may sacrifice accuracy. In this work, we are proposing an implementation of ``stairs'' assisted greedy generation. It is a modified assisted generation methodology that makes use of a smaller model's fast generation, large model's batch prediction, and "stairs" validation in order to achieve a speed up in prediction generation. Results show between 9.58 and 17.24 percent inference time reduction compared to a stand-alone large LLM prediction in a text generation task without a loss in accuracy.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 6 of its 12 references that are in the map. First posted on arXiv 29 Jul 2024.

Builds on

6 of 12 references.

References · 6

  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  3. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  4. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  5. 2016Professor Forcing: A New Algorithm for Training Recurrent Networkscited by 214
  6. 2016A Thorough Examination of the CNN/Daily Mail Reading Comprehension Taskcited by 71

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Efficient Inference for Large Language Models
  2. 2024A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models
  3. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
  4. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques
  5. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  6. 2024Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generation
  7. 2023Lightweight reranking for language model generations
  8. 2026Recipes for Steering and Scaling LLMs via Sampling