mlsanityData to 7 Oct 2026

QuickLLaMA: Query-aware Inference Acceleration for Large Language Models

arXiv 2406.07528 ↗First posted 11 Jun 2024cs.LGcited by 4

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The capacity of Large Language Models (LLMs) to comprehend and reason over long contexts is pivotal for advancements in diverse fields. Yet, they still stuggle with capturing long-distance dependencies within sequences to deeply understand semantics. To address this issue, we introduce Query-aware Inference for LLMs (Q-LLM), a system designed to process extensive sequences akin to human cognition. By focusing on memory data relevant to a given query, Q-LLM can accurately capture pertinent information within a fixed window size and provide precise answers to queries. It doesn't require extra training and can be seamlessly integrated with any LLMs. Q-LLM using LLaMA3 (QuickLLaMA) can read Harry Potter within 30s and accurately answer the questions. On widely recognized benchmarks, Q-LLM improved by 7.17% compared to the current state-of-the-art on LLaMA3, and by 3.26% on Mistral on the ∞-bench. In the Needle-in-a-Haystack and BABILong task, Q-LLM improved upon the current SOTA by 7.0% and 6.1%. Our code can be found in https://github.com/dvlab-research/Q-LLM.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 35 of its 56 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

35 of 56 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache5 Jan
  2. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs2 Nov

Other references · 33

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  12. 2020Reformer: The Efficient Transformercited by 1,658
21 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  4. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  6. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  7. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  8. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  9. 2014Neural Turing Machinescited by 630
  10. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  11. 2014Memory Networkscited by 354
  12. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  13. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  14. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  15. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  16. 2019Explicit Sparse Transformer: Concentrated Attention Through Explicit Selectioncited by 61
  17. 2023A Survey on Long Text Modeling with Transformerscited by 47
  18. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 43
  19. 2023TagCLIP: Improving Discrimination Ability of Open-Vocabulary Semantic Segmentationcited by 14
  20. 2023BAL: Balancing Diversity and Novelty for Active Learningcited by 9
  21. 2023MoTCoder: Elevating Large Language Models with Modular of Thought for Challenging Programming Taskscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  2. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
  3. 2025PromptDistill: Query-based Selective Token Retention in Intermediate Layers for Efficient Large Language Model Inferencecited by 1
  4. 2026Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
  2. 2024A Survey on Efficient Inference for Large Language Models
  3. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques
  4. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
  5. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
  6. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  7. 2025Accelerating LLM Inference with Precomputed Query Storage
  8. 2025xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference