mlsanityData to 7 Oct 2026

Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning

arXiv 2310.03094 ↗First posted 4 Oct 2023cs.CL cs.AI cs.LGcited by 140

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) such as GPT-4 have exhibited remarkable performance in a variety of tasks, but this strong performance often comes with the high expense of using paid API services. In this paper, we are motivated to study building an LLM cascade to save the cost of using LLMs, particularly for performing reasoning (e.g., mathematical, causal) tasks. Our cascade pipeline follows the intuition that simpler questions can be addressed by a weaker but more affordable LLM, whereas only the challenging questions necessitate the stronger and more expensive LLM. To realize this decision-making, we consider the "answer consistency" of the weaker LLM as a signal of the question difficulty and propose several methods for the answer sampling and consistency checking, including one leveraging a mixture of two thought representations (i.e., Chain-of-Thought and Program-of-Thought). Through experiments on six reasoning benchmark datasets, with GPT-3.5-turbo and GPT-4 being the weaker and stronger LLMs, respectively, we demonstrate that our proposed LLM cascades can achieve performance comparable to using solely the stronger LLM but require only 40% of its cost.

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 40 of its 55 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

40 of 55 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Automatic Model Selection with Large Language Models for Reasoning23 May
  2. 2022PAL: Program-aided Language Models18 Nov

Other references · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
26 more
  1. 2023Teaching Large Language Models to Self-Debugcited by 963
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  3. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  4. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  5. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  6. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  7. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  8. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  9. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  10. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  11. 2023Large Language Models as Tool Makerscited by 293
  12. 2022Prompting GPT-3 To Be Reliablecited by 258
  13. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  14. 2022Binding Language Models in Symbolic Languagescited by 224
  15. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 166
  16. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  17. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  18. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  19. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 94
  20. 2023Batch Prompting: Efficient Inference with Large Language Model APIscited by 84
  21. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 43
  22. 2022Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutionscited by 38
  23. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 37
  24. 2023Causal Reasoning of Entities and Events in Procedural Textscited by 23
  25. 2023Human-in-the-Loop through Chain-of-Thoughtcited by 20
  26. 2023BatchPrompt: Accomplish more with lesscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 37; 2025: 51; 2026: 48.

140 papers.

Sort:

By area:

  1. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  2. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  3. 2025Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketchingcited by 116
  4. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  5. 2024Large Language Models and Games: A Survey and Roadmapcited by 91
  6. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  7. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  8. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  9. 2024Learning to Route LLMs with Confidence Tokenscited by 66
  10. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 63
  11. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  12. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
38 more
  1. 2025Detecting High-Stakes Interactions with Activation Probescited by 45
  2. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  3. 2024Faster Cascades via Speculative Decodingcited by 38
  4. 2026Building Production-Ready Probes For Geminicited by 38
  5. 2024System-1.x: Learning to Balance Fast and Slow Planning with Language Modelscited by 35
  6. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  7. 2025CARROT: A Cost Aware Rate Optimal Routercited by 34
  8. 2024Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selectioncited by 32
  9. 2024Semantic Operators: A Declarative Model for Rich, AI-based Data Processingcited by 29
  10. 2024Don't Go To Extremes: Revealing the Excessive Sensitivity and Calibration Limitations of LLMs in Implicit Hate Speech Detectioncited by 25
  11. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 25
  12. 2024Steering Large Language Models between Code Execution and Textual Reasoningcited by 24
  13. 2024Integrate the Essence and Eliminate the Dross: Fine-Grained Self-Consistency for Free-Form Language Generationcited by 21
  14. 2024A Survey on the Honesty of Large Language Modelscited by 21
  15. 2025Learning to Reason via Mixture-of-Thought for Logical Reasoningcited by 20
  16. 2024BudgetMLAgent: A Cost-Effective LLM Multi-Agent system for Automating Machine Learning Taskscited by 19
  17. 2023Mixed Distillation Helps Smaller Language Model Better Reasoningcited by 18
  18. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
  19. 2024Agreement-Based Cascading for Efficient Inferencecited by 16
  20. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  21. 2024Cascade-Aware Training of Language Modelscited by 15
  22. 2024Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guaranteescited by 15
  23. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  24. 2024AdaCQR: Enhancing Query Reformulation for Conversational Search via Sparse and Dense Retrieval Alignmentcited by 14
  25. 2025Cost-Saving LLM Cascades with Early Abstentioncited by 14
  26. 2025Beyond Linear Probes: Dynamic Safety Monitoring for Language Modelscited by 14
  27. 2024Synergy-of-Thoughts: Eliciting Efficient Reasoning in Hybrid Language Modelscited by 13
  28. 2024CausalEval: Towards Better Causal Reasoning in Language Modelscited by 13
  29. 2024PickLLM: Context-Aware RL-Assisted Large Language Model Routingcited by 13
  30. 2025Efficient Training-Free Online Routing for High-Volume Multi-LLM Servingcited by 13
  31. 2025RadialRouter: Structured Representation for Efficient and Robust Large Language Models Routingcited by 11
  32. 2025Gatekeeper: Improving Model Cascades Through Confidence Tuningcited by 10
  33. 2025Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Modelscited by 10
  34. 2024Enabling Weak LLMs to Judge Response Reliability via Meta Rankingcited by 9
  35. 2025Rational Tuning of LLM Cascades via Probabilistic Modelingcited by 9
  36. 2025TagRouter: Learning Route to LLMs through Tags for Open-Domain Text Generation Taskscited by 9
  37. 2025Don't Throw Away Your Pretrained Modelcited by 9
  38. 2026Latent Space Communication via K-V Cache Alignmentcited by 9

50 most cited of 140 All 140 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Tandem: Riding Together with Large and Small Language Models for Efficient Reasoning
  2. 2024Refining Answer Distributions for Improved Large Language Model Reasoning
  3. 2022Faithful Reasoning Using Large Language Models
  4. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  5. 2026Maistros: A Greek Large Language Model Adapted Through Knowledge Distillation From Large Reasoning Models
  6. 2024Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling
  7. 2024CLR-Bench: Evaluating Large Language Models in College-level Reasoning
  8. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance