mlsanityData to 14 Oct 2026

Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inference

arXiv 2601.22132 ↗First posted 29 Jan 2026cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) deliver state-of-the-art performance on complex reasoning tasks, but their inference costs limit deployment at scale. Small Language Models (SLMs) offer dramatic cost savings yet lag substantially in accuracy. Existing approaches - routing and cascading - treat the LLM as an all-or-nothing resource: either the query bypasses the LLM entirely, or the LLM generates a complete response at full cost. We introduce LLM Shepherding, a framework that requests only a short prefix (a hint) from the LLM and provides it to SLM. This simple mechanism is surprisingly effective for math and coding tasks: even hints comprising 10-30% of the full LLM response improve SLM accuracy significantly. Shepherding generalizes both routing and cascading, and it achieves lower cost under oracle decision-making. We develop a two-stage predictor that jointly determines whether a hint is needed and how many tokens to request. On the widely-used mathematical reasoning (GSM8K, CNK12) and code generation (HumanEval, MBPP) benchmarks, Shepherding reduces costs by 42-94% relative to LLM-only inference. Compared to state-of-the-art routing and cascading baselines, shepherding delivers up to 2.8x cost reduction while matching accuracy. To our knowledge, this is the first work to exploit token-level budget control for SLM-LLM collaboration.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 26 of its 31 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

26 of 31 references.

References · 26

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  7. 2020Knowledge Distillation: A Surveycited by 1,603
  8. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  10. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  11. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  12. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
14 more
  1. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  2. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  3. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  4. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 165
  5. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
  6. 2023AutoMix: Automatically Mixing Language Modelscited by 125
  7. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 121
  8. 2023Online Speculative Decodingcited by 113
  9. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  10. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 71
  11. 2024Agreement-Based Cascading for Efficient Inferencecited by 16
  12. 2019Multi-Sample Dropout for Accelerated Training and Better Generalizationcited by 14
  13. 2025C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generationcited by 9
  14. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026FutureBridge: Token Selection Beyond Local Preference in Collaborative Decodingcited by 1
  2. 2026What Does a Token Cost? A Mixture-of-Agents Measurement of Sufficient Per-Token Computecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Inference Scaling vs Reasoning: An Empirical Analysis of Compute-Optimal LLM Problem-Solving
  2. 2025Guiding Reasoning in Small Language Models with LLM Assistance
  3. 2026PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference
  4. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  5. 2025Bridging the Reasoning Gap: Small LLMs Can Plan with Generalised Strategies
  6. 2025Scalable LLM Reasoning Acceleration with Low-rank Distillation
  7. 2025Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning
  8. 2026Tandem: Riding Together with Large and Small Language Models for Efficient Reasoning