# OrchestraLLM: Efficient Orchestration of Language Models for Dialogue State Tracking

arXiv: https://arxiv.org/abs/2311.09758 · HTML: https://mlsanity.com/paper/2311.09758/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 58 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 30 of 39)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT](https://mlsanity.com/paper/1909.05840/) (2019)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://mlsanity.com/paper/2201.05966/) (2022)
- [FastBERT: a Self-distilling BERT with Adaptive Inference Time](https://mlsanity.com/paper/2004.02178/) (2020)
- [Structured Pruning of Large Language Models](https://mlsanity.com/paper/1910.04732/) (2019)
- [Block Pruning For Faster Transformers](https://mlsanity.com/paper/2109.04838/) (2021)
- [Structured Pruning Learns Compact and Accurate Models](https://mlsanity.com/paper/2204.00408/) (2022)
- [Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System](https://mlsanity.com/paper/2109.14739/) (2021)
- [Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling](https://mlsanity.com/paper/2308.06077/) (2023)
- [Dialogue State Tracking with a Language Model using Schema-Driven Prompting](https://mlsanity.com/paper/2109.07506/) (2021)
- [MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation](https://mlsanity.com/paper/2104.00773/) (2021)
- [Exploring the Benefits of Training Expert Language Models over Instruction Tuning](https://mlsanity.com/paper/2302.03202/) (2023)
- [Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search](https://mlsanity.com/paper/2010.07003/) (2020)
- [SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching](https://mlsanity.com/paper/2005.05298/) (2020)

## Built on by (most cited first, 19 of 19)

- [Large Language Models Meet NLP: A Survey](https://mlsanity.com/paper/2405.12819/) (2024)
- [Universal Model Routing for Efficient LLM Inference](https://mlsanity.com/paper/2502.08773/) (2025)
- [RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs](https://mlsanity.com/paper/2503.10657/) (2025)
- [Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems](https://mlsanity.com/paper/2502.00409/) (2025)
- [Agreement-Based Cascading for Efficient Inference](https://mlsanity.com/paper/2407.02348/) (2024)
- [Cost-Aware Contrastive Routing for LLMs](https://mlsanity.com/paper/2508.12491/) (2025)
- [Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving](https://mlsanity.com/paper/2509.02718/) (2025)
- [TagRouter: Learning Route to LLMs through Tags for Open-Domain Text Generation Tasks](https://mlsanity.com/paper/2506.12473/) (2025)
- [Arch-Router: Aligning LLM Routing with Human Preferences](https://mlsanity.com/paper/2506.16655/) (2025)
- [Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Models](https://mlsanity.com/paper/2410.10846/) (2024)
- [Efficiently Deploying LLMs with Controlled Risk](https://mlsanity.com/paper/2410.02173/) (2024)
- [Picking the Right Specialist: Attentive Neural Process-based Selection of Task-Specialized Models as Tools for Agentic Healthcare Systems](https://mlsanity.com/paper/2602.14901/) (2026)
- [CorrectionLM: Self-Corrections with SLM for Dialogue State Tracking](https://mlsanity.com/paper/2410.18209/) (2024)
- [PIE: Performance Interval Estimation for Free-Form Generation Tasks](https://mlsanity.com/paper/2509.07309/) (2025)
- [Agentic Persona Control and Task State Tracking for Realistic User Simulation in Interactive Scenarios](https://mlsanity.com/paper/2601.15290/) (2025)
- [Optimal Splitting of Language Models from Mixtures to Specialized Domains](https://mlsanity.com/paper/2603.19149/) (2026)
- [Capturing LLM Capabilities via Evidence-Calibrated Query Clustering](https://mlsanity.com/paper/2605.17110/) (2026)
- [Token-Operations-Oriented Inference Optimization Techniques for Large Models](https://mlsanity.com/paper/2606.20295/) (2026)
- [The Hidden States Cookbook: A Large-Scale Ablation Study for Noise-Robust Conversational Intent Classification in Industry](https://mlsanity.com/paper/2610.05394/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have revolutionized the landscape of Natural Language Processing systems, but are computationally expensive. To reduce the cost without sacrificing performance, previous studies have explored various approaches to harness the potential of Small Language Models (SLMs) as cost-effective alternatives to their larger counterparts. Driven by findings that SLMs and LLMs exhibit complementary strengths in a structured knowledge extraction task, this work presents a novel SLM/LLM routing framework designed to improve computational efficiency and enhance task performance. First, exemplar pools are created to represent the types of contexts where each LM provides a more reliable answer, leveraging a sentence embedding fine-tuned so that context similarity is close to dialogue state similarity. Then, during inference, the k-nearest exemplars to the testing instance are retrieved, and the instance is routed according to majority vote. In dialogue state tracking tasks, the proposed routing framework enhances performance substantially compared to relying solely on LLMs, while reducing the computational costs by over 50%.