# LLMs on a Budget? Say HOLA

arXiv: https://arxiv.org/abs/2506.18952 · HTML: https://mlsanity.com/paper/2506.18952/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 23 Jun 2025.

## Builds on (most cited first, 12 of 12)

- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [A Careful Examination of Large Language Model Performance on Grade School Arithmetic](https://mlsanity.com/paper/2405.00332/) (2024)
- [Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline](https://mlsanity.com/paper/2305.13144/) (2023)
- [Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation](https://mlsanity.com/paper/2308.10335/) (2023)
- [CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcare](https://mlsanity.com/paper/2312.11541/) (2023)
- [Online Adaptation of Language Models with a Memory of Amortized Contexts](https://mlsanity.com/paper/2403.04317/) (2024)
- [TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction](https://mlsanity.com/paper/2310.15556/) (2023)
- [When to Retrieve: Teaching LLMs to Utilize Information Retrieval Effectively](https://mlsanity.com/paper/2404.19705/) (2024)
- [DataLab: A Unified Platform for LLM-Powered Business Intelligence](https://mlsanity.com/paper/2412.02205/) (2024)
- [Institutional Platform for Secure Self-Service Large Language Model Exploration](https://mlsanity.com/paper/2402.00913/) (2024)
- [Pareto-Optimized Open-Source LLMs for Healthcare via Context Retrieval](https://mlsanity.com/paper/2409.15127/) (2024)
- [Comparative Analysis of Drug-GPT and ChatGPT LLMs for Healthcare Insights: Evaluating Accuracy and Relevance in Patient and HCP Contexts](https://mlsanity.com/paper/2307.16850/) (2023)

## Built on by (most cited first, 1 of 1)

- [Are Large Language Models Economically Viable for Industry Deployment?](https://mlsanity.com/paper/2604.19342/) (2026)

## Abstract (arXiv)

Running Large Language Models (LLMs) on edge devices is constrained by high compute and memory demands posing a barrier for real-time applications in sectors like healthcare, education, and embedded systems. Current solutions such as quantization, pruning, and retrieval-augmented generation (RAG) offer only partial optimizations and often compromise on speed or accuracy. We introduce HOLA, an end-to-end optimization framework for efficient LLM deployment. Internally, it leverages Hierarchical Speculative Decoding (HSD) for faster inference without quality loss. Externally, AdaComp-RAG adjusts retrieval complexity based on context needs. Together with LoBi, which blends structured pruning (LoRA) and quantization, HOLA delivers significant gains: 17.6% EMA on GSM8K, 10.5% MCA on ARC, and reduced latency and memory on edge devices like Jetson Nano--proving both scalable and production-ready.