# The Importance of Encoder Choice:A Tabular-Image Study

arXiv: https://arxiv.org/abs/2607.07756 · HTML: https://mlsanity.com/paper/2607.07756/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 48 references that are in the map.
- First posted on arXiv 8 Jul 2026.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [SAITS: Self-Attention-based Imputation for Time Series](https://mlsanity.com/paper/2202.08516/) (2022)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [TabICLv2: A better, faster, scalable, and open tabular foundation model](https://mlsanity.com/paper/2602.11139/) (2026)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [TabDPT: Scaling Tabular Foundation Models on Real Data](https://mlsanity.com/paper/2410.18164/) (2024)
- [TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling](https://mlsanity.com/paper/2410.24210/) (2024)
- [COVID-19 Image Data Collection: Prospective Predictions Are the Future](https://mlsanity.com/paper/2006.11988/) (2020)
- [XTab: Cross-table Pretraining for Tabular Transformers](https://mlsanity.com/paper/2305.06090/) (2023)
- [Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data](https://mlsanity.com/paper/2303.14080/) (2023)
- [TALENT: A Tabular Analytics and Learning Toolbox](https://mlsanity.com/paper/2407.04057/) (2024)
- [Revisiting Pretraining Objectives for Tabular Deep Learning](https://mlsanity.com/paper/2207.03208/) (2022)
- [A Survey on Table Question Answering: Recent Advances](https://mlsanity.com/paper/2207.05270/) (2022)
- [A Closer Look at Deep Learning Methods on Tabular Datasets](https://mlsanity.com/paper/2407.00956/) (2024)
- [ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics](https://mlsanity.com/paper/2111.13424/) (2021)
- [TIP: Tabular-Image Pre-training for Multimodal Classification with Incomplete Data](https://mlsanity.com/paper/2407.07582/) (2024)
- [UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Science](https://mlsanity.com/paper/2307.09249/) (2023)
- [Table Foundation Models: on knowledge pre-training for tabular learning](https://mlsanity.com/paper/2505.14415/) (2025)
- [DVM-CAR: A large-scale automotive dataset for visual marketing research and applications](https://mlsanity.com/paper/2109.00881/) (2021)

## Abstract (arXiv)

Multimodal learning usually requires a dedicated encoder per modality. When a tabular modality is involved, prior work has been mostly using a plain MLP as the encoder. Yet if it were a strong encoder, the tabular domain would not be ``the last unconquered castle for deep learning''. This study evaluates state-of-the-art tabular models as encoders in the image-tabular setting for the first time. An obstacle stands out. In-Context Learning models, among the best performing methods in the tabular domain, require labels to process instances, making it non-trivial to embed training and test instances the same way. We addressed this problem across multiple models of this family. With this study, we would like to highlight the importance of encoder factor in the multimodal learning.