mlsanityData to 7 Oct 2026

ODIA: Oriented Distillation for Inline Acceleration of LLM-based Function Calling

arXiv 2507.08877 ↗First posted 10 Jul 2025cs.LG cs.AIcited by 0

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Function Calling is a crucial technique that enables Large Language Models (LLMs) to interact with external systems through APIs. However, the high latency associated with LLM-based Function Calling significantly impacts user experience. This paper presents a novel approach called Oriented Distillation for Inline Acceleration (ODIA) that leverages online user interaction data to accelerate Function Calling. By automatically identifying "simple queries" from production traffic and distilling knowledge from larger models to smaller ones, our method reduces response latency by 45% (expected) and 78% (median) while maintaining accuracy. We demonstrate the effectiveness of our approach through real-world deployment in a music application, where the smaller model successfully handles 60% of traffic with negligible accuracy loss. Our method requires minimal human intervention and continuously improves through automated data collection and model updating, making it a practical solution for production environments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 14 of its 18 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

14 of 18 references.

References · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  5. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  6. 2024Learning Harmonized Representations for Speculative Samplingcited by 69
  7. 2024GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Cachingcited by 21
  8. 2024Asynchronous LLM Function Callingcited by 18
  9. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decodingcited by 14
  10. 2024Efficient Prompt Caching via Embedding Similaritycited by 9
  11. 2025SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Modelscited by 9
  12. 2025SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decodingcited by 9
2 more
  1. 2025HAMburger: Accelerating LLM Inference via Token Smashingcited by 4
  2. 2025A Generative Caching System for Large Language Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling
  2. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
  3. 2026Beyond Ideal Instruction: A Comprehensive Framework for Evaluating LLMs in Realistic Interactions
  4. 2024Less is More: Optimizing Function Calling for LLM Execution on Edge Devices
  5. 2024Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems
  6. 2024Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
  8. 2025Harmonia: End-to-End RAG Serving Optimization