# AlignRec: Aligning and Training in Multimodal Recommendations

arXiv: https://arxiv.org/abs/2403.12384 · HTML: https://mlsanity.com/paper/2403.12384/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 22 of 22)

- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)
- [Graph Neural Networks in Recommender Systems: A Survey](https://mlsanity.com/paper/2011.02260/) (2020)
- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts](https://mlsanity.com/paper/2111.02358/) (2021)
- [VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback](https://mlsanity.com/paper/1510.01784/) (2015)
- [Mining Latent Structures for Multimedia Recommendation](https://mlsanity.com/paper/2104.09036/) (2021)
- [GRCN: Graph-Refined Convolutional Network for Multimedia Recommendation with Implicit Feedback](https://mlsanity.com/paper/2111.02036/) (2021)
- [Bootstrap Latent Representations for Multi-modal Recommendation](https://mlsanity.com/paper/2207.05969/) (2022)
- [A Tale of Two Graphs: Freezing and Denoising Graph Structures for Multimodal Recommendation](https://mlsanity.com/paper/2211.06924/) (2022)
- [A Survey of Machine Learning for Computer Architecture and Systems](https://mlsanity.com/paper/2102.07952/) (2021)
- [Towards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Prediction Models](https://mlsanity.com/paper/2209.06053/) (2022)
- [Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization](https://mlsanity.com/paper/2206.04496/) (2022)
- [Multi-View Graph Convolutional Network for Multimedia Recommendation](https://mlsanity.com/paper/2308.03588/) (2023)
- [TorchScale: Transformers at Scale](https://mlsanity.com/paper/2211.13184/) (2022)
- [Augmented Negative Sampling for Collaborative Filtering](https://mlsanity.com/paper/2308.05972/) (2023)

## Built on by (most cited first, 21 of 21)

- [MMSkills: Towards Multimodal Skills for General Visual Agents](https://mlsanity.com/paper/2605.13527/) (2026)
- [On the Value of Cross-Modal Misalignment in Multimodal Representation Learning](https://mlsanity.com/paper/2504.10143/) (2025)
- [Structured Spectral Reasoning for Frequency-Adaptive Multimodal Recommendation](https://mlsanity.com/paper/2512.01372/) (2025)
- [MUSE: A Simple Yet Effective Multimodal Search-Based Framework for Lifelong User Interest Modeling](https://mlsanity.com/paper/2512.07216/) (2025)
- [C-MAG: Cascade Multimodal Attributed Graphs for Supply Chain Link Prediction](https://mlsanity.com/paper/2508.08071/) (2025)
- [Empowering Large Language Model for Sequential Recommendation via Multimodal Embeddings and Semantic IDs](https://mlsanity.com/paper/2509.02017/) (2025)
- [A Metric for MLLM Alignment in Large-scale Recommendation](https://mlsanity.com/paper/2508.04963/) (2025)
- [Multi-modal Adaptive Mixture of Experts for Cold-start Recommendation](https://mlsanity.com/paper/2508.08042/) (2025)
- [MISS: Multi-Modal Tree Indexing and Searching with Lifelong Sequential Behavior for Retrieval Recommendation](https://mlsanity.com/paper/2508.14515/) (2025)
- [DiffCold: A Diffusion-based Generative Model for Cold-Start Item Recommendation](https://mlsanity.com/paper/2606.12245/) (2026)
- [Learning from Unreachable Rewards: Hint-Conditioned Reinforcement Learning for Generative Recommendation](https://mlsanity.com/paper/2608.11980/) (2026)
- [Learning Multi-Modal Mobility Dynamics for Generalized Next Location Recommendation](https://mlsanity.com/paper/2512.22605/) (2025)
- [VLM4Rec: Multimodal Semantic Representation for Recommendation with Large Vision-Language Models](https://mlsanity.com/paper/2603.12625/) (2026)
- [Anchored Alignment: Preventing Positional Collapse in Multimodal Recommender Systems](https://mlsanity.com/paper/2603.12726/) (2026)
- [Fusion and Alignment Enhancement with Large Language Models for Tail-item Sequential Recommendation](https://mlsanity.com/paper/2604.03688/) (2026)
- [TimeMM: Time-as-Operator Spectral Filtering for Dynamic Multimodal Recommendation](https://mlsanity.com/paper/2604.26247/) (2026)
- [Band Together: Untargeted Adversarial Training with Multimodal Coordination against Evasion-based Promotion Attacks](https://mlsanity.com/paper/2605.06238/) (2026)
- [FLUID: From Ephemeral IDs to Multimodal Semantic Codes for Industrial-Scale Livestreaming Recommendation](https://mlsanity.com/paper/2605.21832/) (2026)
- [GALA: Generative Aligned Learning for Adaptive Multimodal Representation in the Taobao Shangou Recommender System](https://mlsanity.com/paper/2607.29213/) (2026)
- [MURAL: Multimodal Uncertainty-aware Recommendation via Adaptive edge Learning](https://mlsanity.com/paper/2609.04574/) (2026)
- [OT-FlowRec: Optimal Transport-Guided Multi-Bridge Flow for Multimodal Sequential Recommendation](https://mlsanity.com/paper/2610.16038/) (2026)

## Abstract (arXiv)

With the development of multimedia systems, multimodal recommendations are playing an essential role, as they can leverage rich contexts beyond interactions. Existing methods mainly regard multimodal information as an auxiliary, using them to help learn ID features; However, there exist semantic gaps among multimodal content features and ID-based features, for which directly using multimodal information as an auxiliary would lead to misalignment in representations of users and items. In this paper, we first systematically investigate the misalignment issue in multimodal recommendations, and propose a solution named AlignRec. In AlignRec, the recommendation objective is decomposed into three alignments, namely alignment within contents, alignment between content and categorical ID, and alignment between users and items. Each alignment is characterized by a specific objective function and is integrated into our multimodal recommendation framework. To effectively train AlignRec, we propose starting from pre-training the first alignment to obtain unified multimodal features and subsequently training the following two alignments together with these features as input. As it is essential to analyze whether each multimodal feature helps in training and accelerate the iteration cycle of recommendation models, we design three new classes of metrics to evaluate intermediate performance. Our extensive experiments on three real-world datasets consistently verify the superiority of AlignRec compared to nine baselines. We also find that the multimodal features generated by AlignRec are better than currently used ones, which are to be open-sourced in our repository https://github.com/sjtulyf123/AlignRec_CIKM24.