mlsanityData to 14 Oct 2026

Inference-Time Distillation: Cost-Efficient Agents Without Fine-Tuning or Manual Prompt Engineering

arXiv 2512.02543 ↗First posted 2 Dec 2025cs.LGcited by 2

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Deploying LLM agents at scale typically requires choosing between quality and cost. Existing cost-reduction approaches fail to preserve agility: the ability to iterate rapidly without human time bottlenecks. Prompt engineering is brittle and slows iteration, while fine-tuning requires multi-day training and commitment to fixed designs; both are impractical for iterative workflows and time-sensitive batch jobs. We demonstrate that established inference-time techniques--dynamic in-context learning and self-consistency cascades--can be leveraged to shift the cost-accuracy Pareto frontier while preserving agility. Practitioners run the teacher on a small task subset to collect demonstrations, then immediately deploy a cheaper student on the remainder. At each step, the system retrieves relevant teacher demonstrations as in-context examples. When multiple student samples agree, we proceed; when they diverge, we fall back to the teacher. This requires no prompt engineering or training. On ALFWorld, we match teacher accuracy at 2.5x lower cost (0.059 to 0.024 per episode). On AppWorld, we achieve 3.5x cost reduction while recovering 79% of teacher accuracy. Our empirical analyses provide guidance on key design choices: teacher database size, demonstration set size, retrieval strategy, and cascade thresholds. These analyses highlight inference-time levers for navigating cost-performance tradeoffs without sacrificing human development speed.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 33 of its 39 references that are in the map. First posted on arXiv 2 Dec 2025.

Builds on

33 of 39 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks1 May
  2. 2024AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents13 Mar
  3. 2024TRAD: Enhancing LLM Agents with Step-Wise Thought Retrieval and Aligned Decision10 Mar
  4. 2024Soft Self-Consistency Improves Language Model Agents20 Feb
  5. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMs19 Oct
  6. 2023FireAct: Toward Language Agent Fine-tuning9 Oct
  7. 2023ExpeL: LLM Agents Are Experiential Learners20 Aug

Other references · 26

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  6. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  7. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  8. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  9. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  10. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  11. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  12. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
14 more
  1. 2022Confident Adaptive Language Modelingcited by 310
  2. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  3. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  4. 2024Many-Shot In-Context Learningcited by 206
  5. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 121
  6. 2024RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agentscited by 80
  7. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  8. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 60
  9. 2020CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascadecited by 47
  10. 2020Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Modelscited by 39
  11. 2025Towards Enterprise-Ready Computer Using Generalist Agentcited by 31
  12. 2024Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuningcited by 21
  13. 2022TangoBERT: Reducing Inference Cost by using Cascaded Architecturecited by 13
  14. 2024Fine-Tuning or Fine-Failing? Debunking Performance Myths in Large Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Cost-Utility Alignment in LLM Agent Trajectories:Profiling,Attribution,Diagnosis,Adaptation,and Evaluationcited by 2
  2. 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LIMBO: Lifelong Inference-Time Memory and Budget Optimization for LLM Agents
  2. 2026Skills on the Fly: Test-Time Adaptive Skill Synthesis for LLM Agents
  3. 2026Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory
  4. 2026Decocted Experience Improves Test-Time Inference in LLM Agents
  5. 2026Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
  6. 2026No Time Like the Present: Agentic Test-Time Training for LLM Agents
  7. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective
  8. 2025AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes