mlsanityData to 7 Oct 2026

Inferflow: an Efficient and Highly Configurable Inference Engine for Large Language Models

arXiv 2401.08294 ↗First posted 16 Jan 2024cs.CLcited by 4

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

We present Inferflow, an efficient and highly configurable inference engine for large language models (LLMs). With Inferflow, users can serve most of the common transformer models by simply modifying some lines in corresponding configuration files, without writing a single line of source code. Compared with most existing inference engines, Inferflow has some key features. First, by implementing a modular framework of atomic build-blocks and technologies, Inferflow is compositionally generalizable to new models. Second, 3.5-bit quantization is introduced in Inferflow as a tradeoff between 3-bit and 4-bit quantization. Third, hybrid model partitioning for multi-GPU inference is introduced in Inferflow to better balance inference speed and throughput than the existing partition-by-layer and partition-by-tensor strategies.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 14 of its 21 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

14 of 21 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2018Hierarchical Neural Story Generationcited by 1,182
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  10. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  12. 20218-bit Optimizers via Block-wise Quantizationcited by 395
2 more
  1. 2022Locally Typical Samplingcited by 108
  2. 2023A Frustratingly Simple Decoding Method for Neural Text Generationcited by 16

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 30
  2. 2025Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmissioncited by 4
  3. 2024Towards Coarse-to-Fine Evaluation of Inference Efficiency for Large Language Modelscited by 3
  4. 2025Binary Neural Networks for Large Language Model: A Surveycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  2. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
  3. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inference
  4. 2023Efficient LLM Inference on CPUs
  5. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
  6. 2024ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models
  7. 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference
  8. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs