# MuG: A Multimodal Classification Benchmark on Game Data with Tabular, Textual, and Visual Fields

arXiv: https://arxiv.org/abs/2302.02978 · HTML: https://mlsanity.com/paper/2302.02978/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 79 references that are in the map.
- First posted on arXiv 6 Feb 2023.

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [GraphSAINT: Graph Sampling Based Inductive Learning Method](https://mlsanity.com/paper/1907.04931/) (2019)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning](https://mlsanity.com/paper/2103.01913/) (2021)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks](https://mlsanity.com/paper/1909.01315/) (2019)
- [A Comprehensive Survey of Deep Learning for Image Captioning](https://mlsanity.com/paper/1810.04020/) (2018)
- [MultiModalQA: Complex Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2104.06039/) (2021)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Cross-Modality Fusion Transformer for Multispectral Object Detection](https://mlsanity.com/paper/2111.00273/) (2021)
- [Integrated multimodal artificial intelligence framework for healthcare applications](https://mlsanity.com/paper/2202.12998/) (2022)
- [Text-to-Table: A New Way of Information Extraction](https://mlsanity.com/paper/2109.02707/) (2021)
- [VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles](https://mlsanity.com/paper/2010.05406/) (2020)
- [Multimodal Machine Learning for Automated ICD Coding](https://mlsanity.com/paper/1810.13348/) (2018)
- [TabGNN: Multiplex Graph Neural Network for Tabular Data Prediction](https://mlsanity.com/paper/2108.09127/) (2021)

## Built on by (most cited first, 6 of 6)

- [Emergency Department Decision Support using Clinical Pseudo-notes](https://mlsanity.com/paper/2402.00160/) (2024)
- [Evaluation and Enhancement of Semantic Grounding in Large Vision-Language Models](https://mlsanity.com/paper/2309.04041/) (2023)
- [MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image](https://mlsanity.com/paper/2605.10616/) (2026)
- [Early Risk Prediction of Pediatric Cardiac Arrest from Electronic Health Records via Multimodal Fused Transformer](https://mlsanity.com/paper/2502.07158/) (2025)
- [Unsupervised Ensemble Learning Through Deep Energy-based Models](https://mlsanity.com/paper/2601.20556/) (2026)
- [On the Geometry of Multimodal Saturation: Riemannian VICReg](https://mlsanity.com/paper/2610.06096/) (2026)

## Abstract (arXiv)

Previous research has demonstrated the advantages of integrating data from multiple sources over traditional unimodal data, leading to the emergence of numerous novel multimodal applications. We propose a multimodal classification benchmark MuG with eight datasets that allows researchers to evaluate and improve their models. These datasets are collected from four various genres of games that cover tabular, textual, and visual modalities. We conduct multi-aspect data analysis to provide insights into the benchmark, including label balance ratios, percentages of missing features, distributions of data within each modality, and the correlations between labels and input modalities. We further present experimental results obtained by several state-of-the-art unimodal classifiers and multimodal classifiers, which demonstrate the challenging and multimodal-dependent properties of the benchmark. MuG is released at https://github.com/lujiaying/MUG-Bench with the data, tutorials, and implemented baselines.