mlsanityData to 14 Oct 2026

LLMs on a Budget? Say HOLA

arXiv 2506.18952 ↗First posted 23 Jun 2025cs.LG cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Running Large Language Models (LLMs) on edge devices is constrained by high compute and memory demands posing a barrier for real-time applications in sectors like healthcare, education, and embedded systems. Current solutions such as quantization, pruning, and retrieval-augmented generation (RAG) offer only partial optimizations and often compromise on speed or accuracy. We introduce HOLA, an end-to-end optimization framework for efficient LLM deployment. Internally, it leverages Hierarchical Speculative Decoding (HSD) for faster inference without quality loss. Externally, AdaComp-RAG adjusts retrieval complexity based on context needs. Together with LoBi, which blends structured pruning (LoRA) and quantization, HOLA delivers significant gains: 17.6% EMA on GSM8K, 10.5% MCA on ARC, and reduced latency and memory on edge devices like Jetson Nano--proving both scalable and production-ready.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 12 of its 20 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

12 of 20 references.

References · 12

  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  2. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  3. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  4. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 73
  5. 2023CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcarecited by 43
  6. 2024Online Adaptation of Language Models with a Memory of Amortized Contextscited by 35
  7. 2023TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reductioncited by 34
  8. 2024When to Retrieve: Teaching LLMs to Utilize Information Retrieval Effectivelycited by 21
  9. 2024DataLab: A Unified Platform for LLM-Powered Business Intelligencecited by 8
  10. 2024Institutional Platform for Secure Self-Service Large Language Model Explorationcited by 2
  11. 2024Pareto-Optimized Open-Source LLMs for Healthcare via Context Retrievalcited by 2
  12. 2023Comparative Analysis of Drug-GPT and ChatGPT LLMs for Healthcare Insights: Evaluating Accuracy and Relevance in Patient and HCP Contextscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Are Large Language Models Economically Viable for Industry Deployment?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving
  2. 2026E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments
  3. 2025A Systematic Evaluation of On-Device LLMs: Quantization, Performance, and Resources
  4. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  5. 2024New Solutions on LLM Acceleration, Optimization, and Application
  6. 2023Accelerating LLM Inference with Staged Speculative Decoding
  7. 2024Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devices
  8. 2024On-Device Language Models: A Comprehensive Review