# C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoning

arXiv: https://arxiv.org/abs/2511.07396 · HTML: https://mlsanity.com/paper/2511.07396/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 14 of its 54 references that are in the map.
- First posted on arXiv 10 Nov 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 14 of 14)

- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [Mixture-of-Agents Enhances Large Language Model Capabilities](https://mlsanity.com/paper/2406.04692/) (2024)
- [O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning](https://mlsanity.com/paper/2501.12570/) (2025)
- [DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Models](https://mlsanity.com/paper/2503.04472/) (2025)
- [Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning](https://mlsanity.com/paper/2503.07572/) (2025)
- [A Unified Approach to Routing and Cascading for LLMs](https://mlsanity.com/paper/2410.10347/) (2024)
- [Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoning](https://mlsanity.com/paper/2408.13457/) (2024)
- [MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMs](https://mlsanity.com/paper/2407.10834/) (2024)
- [Eagle: Efficient Training-Free Router for Multi-LLM Inference](https://mlsanity.com/paper/2409.15518/) (2024)
- [Routoo: Learning to Route to Large Language Models Effectively](https://mlsanity.com/paper/2401.13979/) (2024)
- [Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimization](https://mlsanity.com/paper/2501.17974/) (2025)
- [DynaThink: Fast or Slow? A Dynamic Decision-Making Framework for Large Language Models](https://mlsanity.com/paper/2407.01009/) (2024)
- [Path-Consistency with Prefix Enhancement for Efficient Inference in LLMs](https://mlsanity.com/paper/2409.01281/) (2024)
- [Efficient Dynamic Ensembling for Multiple LLM Experts](https://mlsanity.com/paper/2412.07448/) (2024)

## Built on by (most cited first, 8 of 8)

- [Squeeze Evolve: Unified Multi-Model Orchestration for Verifier-Free Evolution](https://mlsanity.com/paper/2604.07725/) (2026)
- [The Handoff Tax: Continuing Non-Native Trajectories in LLM Agents](https://mlsanity.com/paper/2608.24358/) (2026)
- [Is Escalation Worth It? On the Depth of LLM Cascades](https://mlsanity.com/paper/2605.06350/) (2026)
- [CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents](https://mlsanity.com/paper/2607.19338/) (2026)
- [Proactive Routing to Interpretable Surrogates with Distribution-Free Safety Guarantees](https://mlsanity.com/paper/2603.14623/) (2026)
- [The Capability Frontier: Benchmarks Miss 82% of Model Performance](https://mlsanity.com/paper/2606.26836/) (2026)
- [AutoRelAnnotator: Calibrated Model Cascades for Cost-Efficient Relevance Evaluation in Sponsored Search](https://mlsanity.com/paper/2606.25871/) (2026)
- [Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference](https://mlsanity.com/paper/2607.25018/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have achieved impressive results on complex reasoning tasks, but their high inference cost remains a major barrier to real-world deployment. A promising solution is to use cascaded inference, where small, cheap models handle easy queries, and only the hardest examples are escalated to more powerful models. However, existing cascade methods typically rely on supervised training with labeled data, offer no theoretical generalization guarantees, and provide limited control over test-time computational cost. We introduce C3PO (Cost Controlled Cascaded Prediction Optimization), a self-supervised framework for optimizing LLM cascades under probabilistic cost constraints. By focusing on minimizing regret with respect to the most powerful model (MPM), C3PO avoids the need for labeled data by constructing a cascade using only unlabeled model outputs. It leverages conformal prediction to bound the probability that inference cost exceeds a user-specified budget. We provide theoretical guarantees on both cost control and generalization error, and show that our optimization procedure is effective even with small calibration sets. Empirically, C3PO achieves state-of-the-art performance across a diverse set of reasoning benchmarks including GSM8K, MATH-500, BigBench-Hard and AIME, outperforming strong LLM cascading baselines in both accuracy and cost-efficiency. Our results demonstrate that principled, label-free cascade optimization can enable scalable LLM deployment.