# Movie Recommendation with Poster Attention via Multi-modal Transformer Feature Fusion

arXiv: https://arxiv.org/abs/2407.09157 · HTML: https://mlsanity.com/paper/2407.09157/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 66 references that are in the map.
- First posted on arXiv 12 Jul 2024.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Graph Neural Networks in Recommender Systems: A Survey](https://mlsanity.com/paper/2011.02260/) (2020)
- [Multimodal Token Fusion for Vision Transformers](https://mlsanity.com/paper/2204.08721/) (2022)
- [Collaborative Deep Learning for Recommender Systems](https://mlsanity.com/paper/1409.2944/) (2014)
- [A Scalable Hybrid Research Paper Recommender System for Microsoft Academic](https://mlsanity.com/paper/1905.08880/) (2019)
- [Scientific Paper Recommendation Systems: a Literature Review of recent Publications](https://mlsanity.com/paper/2201.00682/) (2022)
- [Science Concierge: A fast content-based recommendation system for scientific publications](https://mlsanity.com/paper/1604.01070/) (2016)
- [Learning with linear mixed model for group recommendation systems](https://mlsanity.com/paper/2212.08901/) (2022)

## Abstract (arXiv)

Pre-trained models learn general representations from large datsets which can be fine-turned for specific tasks to significantly reduce training time. Pre-trained models like generative pretrained transformers (GPT), bidirectional encoder representations from transformers (BERT), vision transfomers (ViT) have become a cornerstone of current research in machine learning. This study proposes a multi-modal movie recommendation system by extract features of the well designed posters for each movie and the narrative text description of the movie. This system uses the BERT model to extract the information of text modality, the ViT model applied to extract the information of poster/image modality, and the Transformer architecture for feature fusion of all modalities to predict users' preference. The integration of pre-trained foundational models with some smaller data sets in downstream applications capture multi-modal content features in a more comprehensive manner, thereby providing more accurate recommendations. The efficiency of the proof-of-concept model is verified by the standard benchmark problem the MovieLens 100K and 1M datasets. The prediction accuracy of user ratings is enhanced in comparison to the baseline algorithm, thereby demonstrating the potential of this cross-modal algorithm to be applied for movie or video recommendation.