# Multimodal Foundation Model-Driven User Interest Modeling and Behavior Analysis on Short Video Platforms

arXiv: https://arxiv.org/abs/2509.04751 · HTML: https://mlsanity.com/paper/2509.04751/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 5 Sep 2025.

## Builds on (most cited first, 18 of 18)

- [Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Survey](https://mlsanity.com/paper/2412.20367/) (2024)
- [APEER: Automatic Prompt Engineering Enhances Large Language Model Reranking](https://mlsanity.com/paper/2406.14449/) (2024)
- [A Multimodal Fusion Network For Student Emotion Recognition Based on Transformer and Tensor Product](https://mlsanity.com/paper/2403.08511/) (2024)
- [Research on Splicing Image Detection Algorithms Based on Natural Image Statistical Characteristics](https://mlsanity.com/paper/2404.16296/) (2024)
- [Research on reinforcement learning based warehouse robot navigation algorithm in complex warehouse layout](https://mlsanity.com/paper/2411.06128/) (2024)
- [Enhanced Recommendation Combining Collaborative Filtering and Large Language Models](https://mlsanity.com/paper/2412.18713/) (2024)
- [Research on the Design of a Short Video Recommendation System Based on Multimodal Information and Differential Privacy](https://mlsanity.com/paper/2504.08751/) (2025)
- [User Behavior Analysis in Privacy Protection with Large Language Models: A Study on Privacy Preferences with Limited Data](https://mlsanity.com/paper/2505.06305/) (2025)
- [Machine Learning-Based Prediction of Metal-Organic Framework Materials: A Comparative Analysis of Multiple Models](https://mlsanity.com/paper/2507.04493/) (2025)
- [LLM-Driven E-Commerce Marketing Content Optimization: Balancing Creativity and Conversion](https://mlsanity.com/paper/2505.23809/) (2025)
- [Machine Learning based Enterprise Financial Audit Framework and High Risk Identification](https://mlsanity.com/paper/2507.06266/) (2025)
- [An automatic patent literature retrieval system based on LLM-RAG](https://mlsanity.com/paper/2508.14064/) (2025)
- [Research on Personalized Financial Product Recommendation by Integrating Large Language Models and Graph Neural Networks](https://mlsanity.com/paper/2506.05873/) (2025)
- [Research on E-Commerce Long-Tail Product Recommendation Mechanism Based on Large-Scale Language Models](https://mlsanity.com/paper/2506.06336/) (2025)
- [A Deep Learning Algorithm Based on CNN-LSTM Framework for Predicting Cancer Drug Sales Volume](https://mlsanity.com/paper/2506.21927/) (2025)
- [Meta-Learning for Cold-Start Personalization in Prompt-Tuned LLMs](https://mlsanity.com/paper/2507.16672/) (2025)
- [RankFlow: A Multi-Role Collaborative Reranking Workflow Utilizing Large Language Models](https://mlsanity.com/paper/2502.00709/) (2025)
- [Solving Situation Puzzles with Large Language Model and External Reformulation](https://mlsanity.com/paper/2503.18394/) (2025)

## Abstract (arXiv)

With the rapid expansion of user bases on short video platforms, personalized recommendation systems are playing an increasingly critical role in enhancing user experience and optimizing content distribution. Traditional interest modeling methods often rely on unimodal data, such as click logs or text labels, which limits their ability to fully capture user preferences in a complex multimodal content environment. To address this challenge, this paper proposes a multimodal foundation model-based framework for user interest modeling and behavior analysis. By integrating video frames, textual descriptions, and background music into a unified semantic space using cross-modal alignment strategies, the framework constructs fine-grained user interest vectors. Additionally, we introduce a behavior-driven feature embedding mechanism that incorporates viewing, liking, and commenting sequences to model dynamic interest evolution, thereby improving both the timeliness and accuracy of recommendations. In the experimental phase, we conduct extensive evaluations using both public and proprietary short video datasets, comparing our approach against multiple mainstream recommendation algorithms and modeling techniques. Results demonstrate significant improvements in behavior prediction accuracy, interest modeling for cold-start users, and recommendation click-through rates. Moreover, we incorporate interpretability mechanisms using attention weights and feature visualization to reveal the model's decision basis under multimodal inputs and trace interest shifts, thereby enhancing the transparency and controllability of the recommendation system.