mlsanityData to 7 Oct 2026

KnowsTFM: Knowledge-Informed Fine-Tuning of Small Tabular Foundation Models

arXiv 2606.30258 ↗First posted 29 Jun 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Tabular foundation models have advanced deep learning for tabular data by delivering strong default performance across many small and medium tasks. Yet in niche domains, where data is scarce, high-dimensional, and shifted from the pretraining distribution, they may still fail to outperform carefully designed domain-specific methods. Many such domains also provide curated relational knowledge in the form of knowledge graphs and knowledge banks, but how to use this knowledge to improve and steer textitsmall specialist tabular foundation models remains unclear. We address this problem through Knowledge-informed fine-tuning of small Tabular Foundation Models (modelname). Specifically, we study nanoscale TabPFN- and TabICL-style variants, pretrained under controlled synthetic prior families and adapted using two complementary mechanisms: structural attention priors derived from knowledge graphs and parameter-efficient low-rank updates. We show that injecting domain-specific structural knowledge during fine-tuning yields meaningful gains over vanilla variants in specialist settings, whereas gains on general-domain tasks are marginal. We further observe that continual fine-tuning of frontier models can trigger collapse of pretrained knowledge and mechanisms.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 43 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

25 of 43 references.

References · 25

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  4. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  5. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  6. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  7. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  8. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  9. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 267
  10. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  11. 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 170
  12. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
13 more
  1. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  2. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  3. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 97
  4. 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 93
  5. 2024CARTE: Pretraining and Transfer for Tabular Learningcited by 69
  6. 2025Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Modelscited by 67
  7. 2024Retrieval & Fine-Tuning for In-Context Tabular Modelscited by 58
  8. 2024TuneTables: Context Optimization for Scalable Prior-Data Fitted Networkscited by 47
  9. 2022Two-stage LLM Fine-tuning with Less Specialization and More Generalizationcited by 39
  10. 2024KBLaM: Knowledge Base augmented Language Modelcited by 16
  11. 2024KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptationcited by 10
  12. 2025nanoTabPFN: A Lightweight and Educational Reimplementation of TabPFNcited by 10
  13. 2025TabStruct: Measuring Structural Fidelity of Tabular Datacited by 9

Built on by

Not cited yet.