# UMAIR-FPS: User-aware Multi-modal Animation Illustration Recommendation Fusion with Painting Style

arXiv: https://arxiv.org/abs/2402.10381 · HTML: https://mlsanity.com/paper/2402.10381/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Builds on (most cited first, 23 of 23)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems](https://mlsanity.com/paper/1803.05170/) (2018)
- [Product-based Neural Networks for User Response Prediction](https://mlsanity.com/paper/1611.00144/) (2016)
- [Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks](https://mlsanity.com/paper/1708.04617/) (2017)
- [Disentangled Graph Collaborative Filtering](https://mlsanity.com/paper/2007.01764/) (2020)
- [Self-Supervised Multi-Channel Hypergraph Convolutional Network for Social Recommendation](https://mlsanity.com/paper/2101.06448/) (2021)
- [FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction](https://mlsanity.com/paper/1905.09433/) (2019)
- [ConsisRec: Enhancing GNN for Social Recommendation via Consistent Neighbor Aggregation](https://mlsanity.com/paper/2105.02254/) (2021)
- [Multimodal Recommender Systems: A Survey](https://mlsanity.com/paper/2302.03883/) (2023)
- [FinalMLP: An Enhanced Two-Stream MLP Model for CTR Prediction](https://mlsanity.com/paper/2304.00902/) (2023)
- [Deep Reinforcement Learning for List-wise Recommendations](https://mlsanity.com/paper/1801.00209/) (2017)
- [Towards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Prediction Models](https://mlsanity.com/paper/2209.06053/) (2022)
- [Eliminating Gradient Conflict in Reference-based Line-Art Colorization](https://mlsanity.com/paper/2207.06095/) (2022)
- [Aesthetic-based Clothing Recommendation](https://mlsanity.com/paper/1809.05822/) (2018)
- [Brain-Inspired Deep Networks for Image Aesthetics Assessment](https://mlsanity.com/paper/1601.04155/) (2016)
- [Cross-Domain Style Mixing for Face Cartoonization](https://mlsanity.com/paper/2205.12450/) (2022)
- [Using Posters to Recommend Anime and Mangas in a Cold-Start Scenario](https://mlsanity.com/paper/1709.01584/) (2017)
- [Food recognition and recipe analysis: integrating visual content, context and external knowledge](https://mlsanity.com/paper/1801.07239/) (2018)

## Abstract (arXiv)

The rapid advancement of high-quality image generation models based on AI has generated a deluge of anime illustrations. Recommending illustrations to users within massive data has become a challenging and popular task. However, existing anime recommendation systems have focused on text features but still need to integrate image features. In addition, most multi-modal recommendation research is constrained by tightly coupled datasets, limiting its applicability to anime illustrations. We propose the User-aware Multi-modal Animation Illustration Recommendation Fusion with Painting Style (UMAIR-FPS) to tackle these gaps. In the feature extract phase, for image features, we are the first to combine image painting style features with semantic features to construct a dual-output image encoder for enhancing representation. For text features, we obtain text embeddings based on fine-tuning Sentence-Transformers by incorporating domain knowledge that composes a variety of domain text pairs from multilingual mappings, entity relationships, and term explanation perspectives, respectively. In the multi-modal fusion phase, we novelly propose a user-aware multi-modal contribution measurement mechanism to weight multi-modal features dynamically according to user features at the interaction level and employ the DCN-V2 module to model bounded-degree multi-modal crosses effectively. UMAIR-FPS surpasses the stat-of-the-art baselines on large real-world datasets, demonstrating substantial performance enhancements.