# Efficient Hybrid Inference for LLMs: Reward-Based Token Modelling with Selective Cloud Assistance

arXiv: https://arxiv.org/abs/2409.13757 · HTML: https://mlsanity.com/paper/2409.13757/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 3 references that are in the map.
- First posted on arXiv 15 Sep 2024.

## Builds on (most cited first, 3 of 3)

- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [A Long Way to Go: Investigating Length Correlations in RLHF](https://mlsanity.com/paper/2310.03716/) (2023)
- [Preference-grounded Token-level Guidance for Language Model Fine-tuning](https://mlsanity.com/paper/2306.00398/) (2023)

## Built on by (most cited first, 2 of 2)

- [Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques](https://mlsanity.com/paper/2506.06579/) (2025)
- [Large-Small Model Collaboration for Enhancing Edge-Deployed Small Models](https://mlsanity.com/paper/2503.10367/) (2025)

## Abstract (arXiv)

Large language models (LLMs) are known for their exceptional performance across a range of natural language processing tasks, but their deployment comes at a high computational and financial cost. On the other hand, smaller language models (SLMs), which can be deployed on lower-cost edge devices, struggle to match the performance of their larger counterparts. This paper presents a novel hybrid inference approach that leverages the strengths of both model types while minimizing reliance on costly cloud-based LLMs. Unlike existing methods that route entire queries to either an SLM or a cloud LLM, our approach introduces a reward-based mechanism to dynamically determine the involvement of the cloud LLM during token generation. Specifically, each token predicted by the SLM is evaluated against a reward score, and only when this score falls below a certain threshold is the cloud LLM consulted for assistance in the next token prediction. This method not only reduces the traffic to the cloud LLM, thereby lowering costs, but also allows for flexible control over response quality depending on the reward score threshold. Experimental results demonstrate that our approach significantly reduces cloud LLM usage with minimal impact on overall response quality, offering a cost-effective solution for deploying high-performance language models