# Does a Technique for Building Multimodal Representation Matter? -- Comparative Analysis

arXiv: https://arxiv.org/abs/2206.06367 · HTML: https://mlsanity.com/paper/2206.06367/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 39 references that are in the map.
- First posted on arXiv 9 Jun 2022.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers](https://mlsanity.com/paper/2109.04448/) (2021)
- [Efficient Large-Scale Multi-Modal Classification](https://mlsanity.com/paper/1802.02892/) (2018)
- [Imbalanced Big Data Oversampling: Taxonomy, Algorithms, Software, Guidelines and Future Directions](https://mlsanity.com/paper/2107.11508/) (2021)
- [An efficient manifold density estimator for all recommendation systems](https://mlsanity.com/paper/2006.01894/) (2020)
- [Cleora: A Simple, Strong and Scalable Graph Embedding Scheme](https://mlsanity.com/paper/2102.02302/) (2021)
- [Multi-Label Product Categorization Using Multi-Modal Fusion Models](https://mlsanity.com/paper/1907.00420/) (2019)
- [I know why you like this movie: Interpretable Efficient Multimodal Recommender](https://mlsanity.com/paper/2006.09979/) (2020)

## Built on by (most cited first, 22 of 22)

- [Survey of Large Multimodal Model Datasets, Application Categories and Taxonomy](https://mlsanity.com/paper/2412.17759/) (2024)
- [A Review of ChatGPT Applications in Education, Marketing, Software Engineering, and Healthcare: Benefits, Drawbacks, and Research Directions](https://mlsanity.com/paper/2305.00237/) (2023)
- [Enhancing Vision-Language Models with Scene Graphs for Traffic Accident Understanding](https://mlsanity.com/paper/2407.05910/) (2024)
- [MiPa: Mixed Patch Infrared-Visible Modality Agnostic Object Detection](https://mlsanity.com/paper/2404.18849/) (2024)
- [Integrating Large Language Models into a Tri-Modal Architecture for Automated Depression Classification on the DAIC-WOZ](https://mlsanity.com/paper/2407.19340/) (2024)
- [Stock Movement Prediction with Multimodal Stable Fusion via Gated Cross-Attention Mechanism](https://mlsanity.com/paper/2406.06594/) (2024)
- [ARPA: A Novel Hybrid Model for Advancing Visual Word Disambiguation Using Large Language Models and Transformers](https://mlsanity.com/paper/2408.06040/) (2024)
- [Cascaded Cross-Modal Transformer for Audio-Textual Classification](https://mlsanity.com/paper/2401.07575/) (2024)
- [On the Use of Audio to Improve Dialogue Policies](https://mlsanity.com/paper/2410.13385/) (2024)
- [Multi-Scenario Reasoning: Unlocking Cognitive Autonomy in Humanoid Robots for Multimodal Understanding](https://mlsanity.com/paper/2412.20429/) (2024)
- [MMFformer: Multimodal Fusion Transformer Network for Depression Detection](https://mlsanity.com/paper/2508.06701/) (2025)
- [Multimodal AI Systems for Enhanced Laying Hen Welfare Assessment and Productivity Optimization](https://mlsanity.com/paper/2508.07628/) (2025)
- [A Data-to-Product Multimodal Conceptual Framework to Achieve Automated Software Evolution for Context-rich Intelligent Applications](https://mlsanity.com/paper/2404.04821/) (2024)
- [Progressive Alignment with VLM-LLM Feature to Augment Defect Classification for the ASE Dataset](https://mlsanity.com/paper/2404.05183/) (2024)
- [Investigating Brain Connectivity and Regional Statistics from EEG for early stage Parkinson's Classification](https://mlsanity.com/paper/2408.00711/) (2024)
- [Exploring Multi-Modality Dynamics: Insights and Challenges in Multimodal Fusion for Biomedical Tasks](https://mlsanity.com/paper/2411.00725/) (2024)
- [MORAL: A Multimodal Reinforcement Learning Framework for Decision Making in Autonomous Laboratories](https://mlsanity.com/paper/2504.03153/) (2025)
- [Towards a Unified Textual Graph Framework for Spectral Reasoning via Physical and Chemical Information Fusion](https://mlsanity.com/paper/2506.17761/) (2025)
- [Detecting In-Person Conversations in Noisy Real-World Environments with Smartwatch Audio and Motion Sensing](https://mlsanity.com/paper/2507.12002/) (2025)
- [Multimodal learning of melt pool dynamics in laser powder bed fusion](https://mlsanity.com/paper/2509.03029/) (2025)
- [QCaption: Video Captioning and Q&A through Fusion of Large Multimodal Models](https://mlsanity.com/paper/2601.06566/) (2026)
- [BVFLMSP : Bayesian Vertical Federated Learning for Multimodal Survival with Privacy](https://mlsanity.com/paper/2604.02248/) (2026)

## Abstract (arXiv)

Creating a meaningful representation by fusing single modalities (e.g., text, images, or audio) is the core concept of multimodal learning. Although several techniques for building multimodal representations have been proven successful, they have not been compared yet. Therefore it has been ambiguous which technique can be expected to yield the best results in a given scenario and what factors should be considered while choosing such a technique. This paper explores the most common techniques for building multimodal data representations -- the late fusion, the early fusion, and the sketch, and compares them in classification tasks. Experiments are conducted on three datasets: Amazon Reviews, MovieLens25M, and MovieLens1M datasets. In general, our results confirm that multimodal representations are able to boost the performance of unimodal models from 0.919 to 0.969 of accuracy on Amazon Reviews and 0.907 to 0.918 of AUC on MovieLens25M. However, experiments on both MovieLens datasets indicate the importance of the meaningful input data to the given task. In this article, we show that the choice of the technique for building multimodal representation is crucial to obtain the highest possible model's performance, that comes with the proper modalities combination. Such choice relies on: the influence that each modality has on the analyzed machine learning (ML) problem; the type of the ML task; the memory constraints while training and predicting phase.