# Generalized Context in Cross Attention for Transfer Learning of Disjoint Tabular Data

arXiv: https://arxiv.org/abs/2608.28209 · HTML: https://mlsanity.com/paper/2608.28209/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 37 references that are in the map.
- First posted on arXiv 28 Aug 2026.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [XTab: Cross-table Pretraining for Tabular Transformers](https://mlsanity.com/paper/2305.06090/) (2023)
- [Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning](https://mlsanity.com/paper/2404.09491/) (2024)
- [CARTE: Pretraining and Transfer for Tabular Learning](https://mlsanity.com/paper/2402.16785/) (2024)
- [Transfer Learning with Deep Tabular Models](https://mlsanity.com/paper/2206.15306/) (2022)
- [Transformers generalize differently from information stored in context vs in weights](https://mlsanity.com/paper/2210.05675/) (2022)
- [Solving Inefficiency of Self-supervised Representation Learning](https://mlsanity.com/paper/2104.08760/) (2021)
- [Towards Cross-Table Masked Pretraining for Web Data Mining](https://mlsanity.com/paper/2307.04308/) (2023)
- [Table Foundation Models: on knowledge pre-training for tabular learning](https://mlsanity.com/paper/2505.14415/) (2025)
- [TabRet: Pre-training Transformer-based Tabular Models for Unseen Columns](https://mlsanity.com/paper/2303.15747/) (2023)
- [Deep Clustering of Tabular Data by Weighted Gaussian Distribution Learning](https://mlsanity.com/paper/2301.00802/) (2023)
- [Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarking](https://mlsanity.com/paper/2401.04266/) (2024)
- [Are deep learning models superior for missing data imputation in large surveys? Evidence from an empirical comparison](https://mlsanity.com/paper/2103.09316/) (2021)
- [Perturbation of Deep Autoencoder Weights for Model Compression and Classification of Tabular Data](https://mlsanity.com/paper/2205.08358/) (2022)
- [Transfer Learning of Tabular Data by Finetuning Large Language Models](https://mlsanity.com/paper/2501.06863/) (2025)
- [LATTLE: LLM Attention Transplant for Transfer Learning of Tabular Data Across Disparate Domains](https://mlsanity.com/paper/2511.06161/) (2025)
- [StochCA: A Novel Approach for Exploiting Pretrained Models with Cross-Attention](https://mlsanity.com/paper/2402.16092/) (2024)
- [Why pre-training is beneficial for downstream classification tasks?](https://mlsanity.com/paper/2410.08455/) (2024)

## Abstract (arXiv)

Unlike images and text, applying transfer learning to tabular data is challenging due to heterogeneity in feature types, structures, and semantics across disparate domains. Existing methods assume shared features across data tables to enable knowledge transfer between domains, which is unrealistic in practice. mdsThis paper introduces generalized context learning to remove the requirement of shared features across domains. The generalized context captured by transformer projection weights for key, value, and query provides rule-based generalization rather than the domain-specific context conventionally learned from transformer activations. Projection weights for key from the source domain interact with the weight for query in the target domain to achieve Cross-domain Attention Transfer Learning (CATTLE) in a data-agnostic manner. Our experiments on ten pairs of disjoint source-target data sets show that CATTLE can learn generalized context from a single source data set and is rank-wise and statistically superior to nine state-of-the-art baselines, including machine learning, deep learning, and transfer learning methods using large-scale pre-trained models. CATTLE achieves the best average rank (2.9) and delivers a 3.7% average AUROC gain over the baseline methods. The CATTLE source code is available at https://tinyurl.com/pr5s8ywn.