mlsanityData to 7 Oct 2026

Code and Pixels: Multi-Modal Contrastive Pre-training for Enhanced Tabular Data Analysis

arXiv 2501.07304 ↗First posted 13 Jan 2025cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Learning from tabular data is of paramount importance, as it complements the conventional analysis of image and video data by providing a rich source of structured information that is often critical for comprehensive understanding and decision-making processes. We present Multi-task Contrastive Masked Tabular Modeling (MT-CMTM), a novel method aiming to enhance tabular models by leveraging the correlation between tabular data and corresponding images. MT-CMTM employs a dual strategy combining contrastive learning with masked tabular modeling, optimizing the synergy between these data modalities. Central to our approach is a 1D Convolutional Neural Network with residual connections and an attention mechanism (1D-ResNet-CBAM), designed to efficiently process tabular data without relying on images. This enables MT-CMTM to handle purely tabular data for downstream tasks, eliminating the need for potentially costly image acquisition and processing. We evaluated MT-CMTM on the DVM car dataset, which is uniquely suited for this particular scenario, and the newly developed HIPMP dataset, which connects membrane fabrication parameters with image data. Our MT-CMTM model outperforms the proposed tabular 1D-ResNet-CBAM, which is trained from scratch, achieving a relative 1.48% improvement in relative MSE on HIPMP and a 2.38% increase in absolute accuracy on DVM. These results demonstrate MT-CMTM's robustness and its potential to advance the field of multi-modal learning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 71 references that are in the map. First posted on arXiv 13 Jan 2025.

Builds on

35 of 71 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  9. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  12. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
23 more
  1. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  2. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  3. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  4. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 395
  5. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  6. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  7. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  8. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 200
  9. 2022Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detectioncited by 130
  10. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 122
  11. 2019Self-Supervised Generalisation with Meta Auxiliary Learningcited by 88
  12. 2023Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Datacited by 68
  13. 2023DocFormerv2: Local Features for Document Understandingcited by 46
  14. 2021ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Geneticscited by 35
  15. 2021M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretrainingcited by 26
  16. 2022OLIVES Dataset: Ophthalmic Labels for Investigating Visual Eye Semanticscited by 23
  17. 2022Self-supervised Transformer for Deepfake Detectioncited by 22
  18. 2023Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-trainingcited by 22
  19. 2021DVM-CAR: A large-scale automotive dataset for visual marketing research and applicationscited by 17
  20. 2021CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoisingcited by 17
  21. 2021FocusFace: Multi-task Contrastive Learning for Masked Face Recognitioncited by 12
  22. 2023Multimodal Contrastive Learning and Tabular Attention for Automated Alzheimer's Disease Predictioncited by 12
  23. 2022Improvements to Self-Supervised Representation Learning for Masked Image Modelingcited by 1

Built on by

Not cited yet.