# ContentCTR: Frame-level Live Streaming Click-Through Rate Prediction with Multimodal Transformer

arXiv: https://arxiv.org/abs/2306.14392 · HTML: https://mlsanity.com/paper/2306.14392/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 26 Jun 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction](https://mlsanity.com/paper/2006.05639/) (2020)
- [Behavior Sequence Transformer for E-commerce Recommendation in Alibaba](https://mlsanity.com/paper/1905.06874/) (2019)
- [AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction](https://mlsanity.com/paper/2003.11235/) (2020)
- [User Behavior Retrieval for Click-Through Rate Prediction](https://mlsanity.com/paper/2005.14171/) (2020)
- [Video-Text Representation Learning via Differentiable Weak Temporal Alignment](https://mlsanity.com/paper/2203.16784/) (2022)
- [Adaptive Video Highlight Detection by Learning from User History](https://mlsanity.com/paper/2007.09598/) (2020)
- [PHD-GIFs: Personalized Highlight Detection for Automatic GIF Creation](https://mlsanity.com/paper/1804.06604/) (2018)
- [Deep Spatio-Temporal Neural Networks for Click-Through Rate Prediction](https://mlsanity.com/paper/1906.03776/) (2019)
- [Click-Through Rate Prediction with Multi-Modal Hypergraphs](https://mlsanity.com/paper/2109.02398/) (2021)
- [Leveraging Tripartite Interaction Information from Live Stream E-Commerce for Improving Product Recommendation](https://mlsanity.com/paper/2106.03415/) (2021)
- [PR-Net: Preference Reasoning for Personalized Video Highlight Detection](https://mlsanity.com/paper/2109.01799/) (2021)
- [Show Me What I Like: Detecting User-Specific Video Highlights Using Content-Based Multi-Head Attention](https://mlsanity.com/paper/2207.08352/) (2022)

## Built on by (most cited first, 2 of 2)

- [NeSHFS: Neighborhood Search with Heuristic-based Feature Selection for Click-Through Rate Prediction](https://mlsanity.com/paper/2409.08703/) (2024)
- [PEARL: Unbiased Percentile Estimation via Contrastive Learning for Industrial-Scale Livestream Recommendation](https://mlsanity.com/paper/2605.21752/) (2026)

## Abstract (arXiv)

In recent years, live streaming platforms have gained immense popularity as they allow users to broadcast their videos and interact in real-time with hosts and peers. Due to the dynamic changes of live content, accurate recommendation models are crucial for enhancing user experience. However, most previous works treat the live as a whole item and explore the Click-through-Rate (CTR) prediction framework on item-level, neglecting that the dynamic changes that occur even within the same live room. In this paper, we proposed a ContentCTR model that leverages multimodal transformer for frame-level CTR prediction. First, we present an end-to-end framework that can make full use of multimodal information, including visual frames, audio, and comments, to identify the most attractive live frames. Second, to prevent the model from collapsing into a mediocre solution, a novel pairwise loss function with first-order difference constraints is proposed to utilize the contrastive information existing in the highlight and non-highlight frames. Additionally, we design a temporal text-video alignment module based on Dynamic Time Warping to eliminate noise caused by the ambiguity and non-sequential alignment of visual and textual information. We conduct extensive experiments on both real-world scenarios and public datasets, and our ContentCTR model outperforms traditional recommendation models in capturing real-time content changes. Moreover, we deploy the proposed method on our company platform, and the results of online A/B testing further validate its practical significance.