# KnowsTFM: Knowledge-Informed Fine-Tuning of Small Tabular Foundation Models

arXiv: https://arxiv.org/abs/2606.30258 · HTML: https://mlsanity.com/paper/2606.30258/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Builds on (most cited first, 25 of 25)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?](https://mlsanity.com/paper/2405.05904/) (2024)
- [GreaseLM: Graph REASoning Enhanced Language Models for Question Answering](https://mlsanity.com/paper/2201.08860/) (2022)
- [Deep Bidirectional Language-Knowledge Graph Pretraining](https://mlsanity.com/paper/2210.09338/) (2022)
- [Understanding Catastrophic Forgetting in Language Models via Implicit Inference](https://mlsanity.com/paper/2309.10105/) (2023)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [TabDPT: Scaling Tabular Foundation Models on Real Data](https://mlsanity.com/paper/2410.18164/) (2024)
- [Revisiting Catastrophic Forgetting in Large Language Model Tuning](https://mlsanity.com/paper/2406.04836/) (2024)
- [Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models](https://mlsanity.com/paper/2510.21204/) (2025)
- [CARTE: Pretraining and Transfer for Tabular Learning](https://mlsanity.com/paper/2402.16785/) (2024)
- [Retrieval & Fine-Tuning for In-Context Tabular Models](https://mlsanity.com/paper/2406.05207/) (2024)
- [TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks](https://mlsanity.com/paper/2402.11137/) (2024)
- [Two-stage LLM Fine-tuning with Less Specialization and More Generalization](https://mlsanity.com/paper/2211.00635/) (2022)
- [KBLaM: Knowledge Base augmented Language Model](https://mlsanity.com/paper/2410.10450/) (2024)
- [nanoTabPFN: A Lightweight and Educational Reimplementation of TabPFN](https://mlsanity.com/paper/2511.03634/) (2025)
- [KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation](https://mlsanity.com/paper/2403.14950/) (2024)
- [TabStruct: Measuring Structural Fidelity of Tabular Data](https://mlsanity.com/paper/2509.11950/) (2025)

## Abstract (arXiv)

Tabular foundation models have advanced deep learning for tabular data by delivering strong default performance across many small and medium tasks. Yet in niche domains, where data is scarce, high-dimensional, and shifted from the pretraining distribution, they may still fail to outperform carefully designed domain-specific methods. Many such domains also provide curated relational knowledge in the form of knowledge graphs and knowledge banks, but how to use this knowledge to improve and steer textitsmall specialist tabular foundation models remains unclear. We address this problem through Knowledge-informed fine-tuning of small Tabular Foundation Models (modelname). Specifically, we study nanoscale TabPFN- and TabICL-style variants, pretrained under controlled synthetic prior families and adapted using two complementary mechanisms: structural attention priors derived from knowledge graphs and parameter-efficient low-rank updates. We show that injecting domain-specific structural knowledge during fine-tuning yields meaningful gains over vanilla variants in specialist settings, whereas gains on general-domain tasks are marginal. We further observe that continual fine-tuning of frontier models can trigger collapse of pretrained knowledge and mechanisms.