mlsanityData to 14 Oct 2026

TACTICL: Task-Aware Compression of Tabular ICL Models

arXiv 2608.10837 ↗First posted 11 Aug 2026cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

The strong performance of foundation models for tabular tasks comes at substantial inference costs. Distilling models into task-specific architectures reduces model size and computational demands but also sacrifices in-context adaptability. Here we introduce TACTICL, an automated task-aware compression framework for tabular in-context learning models that jointly prunes transformer layers and replaces them with lightweight adapters trained on downstream tasks, thus blending in-context with in-weight learning. We study TACTICL on 47 benchmark datasets and show that we can substitute up to 85% of layers without substantial performance drop on a given downstream task. We further show that TACTICL maintains robustness to data shifts, leaving its in-context ability intact. Overall, TACTICL provides a robust framework for exploiting the depth-wise redundancy of tabular foundation models by combining task-specific adaptation and structured compression. We provide the code at: https://github.com/Hebog/tfm_compression

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 28 of its 44 references that are in the map. First posted on arXiv 11 Aug 2026.

Builds on

28 of 44 references.

References · 28

  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  2. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  3. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  4. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  5. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  6. 2025TabArena: A Living Benchmark for Machine Learning on Tabular Datacited by 176
  7. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  8. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  9. 2026TabICLv2: A better, faster, scalable, and open tabular foundation modelcited by 144
  10. 2025TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Modelscited by 132
  11. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  12. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 76
16 more
  1. 2024TALENT: A Tabular Analytics and Learning Toolboxcited by 52
  2. 2025Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Datacited by 41
  3. 2024Streamlining Redundant Layers to Compress Large Language Modelscited by 35
  4. 2025Scaling Laws for Forgetting during Finetuning with Pretraining Data Injectioncited by 35
  5. 2024Differential learning kinetics govern the transition from memorization to generalization during in-context learningcited by 21
  6. 2025Distilling to Hybrid Attention Models via KL-Guided Layer Selectioncited by 20
  7. 2025TabPFN-Wide: Continued Pre-Training for Extreme Feature Countscited by 16
  8. 2025ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearizationcited by 13
  9. 2025On Finetuning Tabular Foundation Modelscited by 13
  10. 2025Early Stopping Tabular In-Context Learningcited by 12
  11. 2026Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Modelscited by 10
  12. 2026Exploring Fine-Tuning for Tabular Foundation Modelscited by 9
  13. 2026TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attentioncited by 8
  14. 2024TOAST: Transformer Optimization using Adaptive and Simple Transformationscited by 3
  15. 2025Towards Understanding Layer Contributions in Tabular In-Context Learning Modelscited by 1
  16. 2025Light-Weight Benchmarks Reveal the Hidden Hardware Cost of Zero-Shot Tabular Foundation Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026LoopICL: Looping a single transformer block to solve tabular taskscited by 0
  2. 2026Closing the Context Gap: Activation Alignment for Tabular In-Context Learningcited by 0
  3. 2026Tabular Foundation Models Are Effectively Shallowcited by 0
  4. 2026Recurrent In-Context Learning for Tabular Foundation Modelscited by 0