mlsanityData to 7 Oct 2026

Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Data

arXiv 2412.16243 ↗First posted 19 Dec 2024cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper studies the best practices for automatic machine learning (AutoML). While previous AutoML efforts have predominantly focused on unimodal data, the multimodal aspect remains under-explored. Our study delves into classification and regression problems involving flexible combinations of image, text, and tabular data. We curate a benchmark comprising 22 multimodal datasets from diverse real-world applications, encompassing all 4 combinations of the 3 modalities. Across this benchmark, we scrutinize design choices related to multimodal fusion strategies, multimodal data augmentation, converting tabular data into text, cross-modal alignment, and handling missing modalities. Through extensive experimentation and analysis, we distill a collection of effective strategies and consolidate them into a unified pipeline, achieving robust performance on diverse datasets.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 76 of its 104 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

76 of 104 references.

References · 76

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
38 more
  1. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2017Decoupled Weight Decay Regularizationcited by 1,722
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  13. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  14. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  15. 2017Mixed Precision Trainingcited by 1,135
  16. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  17. 2023A Survey on Multimodal Large Language Modelscited by 898
  18. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  19. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  20. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  21. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  22. 2014OpenML: networked science in machine learningcited by 523
  23. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  24. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  25. 2022Multimodal Learning with Transformers: A Surveycited by 399
  26. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 368
  27. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  28. 2018Deep Audio-Visual Speech Recognitioncited by 318
  29. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  30. 2023Multimodal Deep Learningcited by 257
  31. 2019Fast AutoAugmentcited by 243
  32. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
  33. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 225
  34. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  35. 2020Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learningcited by 174
  36. 2018Model Evaluation, Model Selection, and Algorithm Selection in Machine Learningcited by 157
  37. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  38. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fieldscited by 15
  2. 2026MultiModalPFN: Extending Prior-Data Fitted Networks for Multimodal Tabular Learningcited by 9
  3. 2025Multimodal Representation Learning and Fusioncited by 3
  4. 2026MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Imagecited by 2
  5. 2026The Importance of Encoder Choice:A Tabular-Image Studycited by 0
  6. 2026Walkable to Whom? Capturing Subjective Variability in Walkability Perception Using Multimodal Deep Learningcited by 0