mlsanityData to 14 Oct 2026

Distilling Foundation Models for Agentic What-If Reasoning:Cost, Latency, and Governance in a Hybrid LLM+SLM Architecture

arXiv 2609.16091 ↗First posted 14 Sep 2026cs.LGcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Tabular foundation models deliver strong zero-training predictive performance via in-context learning, but their high inference latency makes them impractical as hot-path decision backends in interactive agentic loops. We distill a TabPFN teacher into a compact feed-forward student across a business-decision simulation on UCI Adult and five OpenML benchmarks: the classification head compresses 53.2M parameters to 8,546 (6,220x); the deployed two-head loan pipeline compresses 111.4M parameters to 17,059 (6,532x). The student retains 95.4-100.5% accuracy and 96.8-100.0% AUC, with the lowest accuracy retention on credit-g at 95.4%; an alpha = 0 hard-label control shows that the teacher's soft targets provide a 2.1-7.0 AUC point gain.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 18 of its 37 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

18 of 37 references.

References · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  8. 2020Knowledge Distillation: A Surveycited by 1,603
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  10. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  11. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  12. 2013Do Deep Nets Really Need to be Deep?cited by 474
6 more
  1. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 418
  2. 2017OpenML Benchmarking Suitescited by 178
  3. 2020Characterising Bias in Compressed Modelscited by 167
  4. 2023Fairlearn: Assessing and Improving Fairness of AI Systemscited by 74
  5. 2024TuneTables: Context Optimization for Scalable Prior-Data Fitted Networkscited by 47
  6. 2024In-Context Data Distillation with TabPFNcited by 19

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Distillation of Tabular Foundation Models into Efficient Predictors
  2. 2025Structured Agent Distillation for Large Language Model
  3. 2025Distilling LLM Agent into Small Models with Retrieval and Code Tools
  4. 2023A Survey of Reasoning with Foundation Models
  5. 2025Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning
  6. 2025Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
  7. 2026SkillTFM: Gated Skill Evolution for Training-Free Adaptation of Tabular Foundation Models
  8. 2024Position: Foundation Agents as the Paradigm Shift for Decision Making