# Advancing Re-Ranking with Multimodal Fusion and Target-Oriented Auxiliary Tasks in E-Commerce Search

arXiv: https://arxiv.org/abs/2408.05751 · HTML: https://mlsanity.com/paper/2408.05751/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 11 Aug 2024.

## Builds on (most cited first, 21 of 21)

- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems](https://mlsanity.com/paper/1803.05170/) (2018)
- [Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction](https://mlsanity.com/paper/2006.05639/) (2020)
- [Recurrent Neural Networks with Top-k Gains for Session-based Recommendations](https://mlsanity.com/paper/1706.03847/) (2017)
- [Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited](https://mlsanity.com/paper/2303.13835/) (2023)
- [Learning a Deep Listwise Context Model for Ranking Refinement](https://mlsanity.com/paper/1804.05936/) (2018)
- [Seq2Slate: Re-ranking and Slate Optimization with RNNs](https://mlsanity.com/paper/1810.02019/) (2018)
- [Sampling Is All You Need on Modeling Long-Term User Behaviors for CTR Prediction](https://mlsanity.com/paper/2205.10249/) (2022)
- [Real-time Short Video Recommendation on Mobile Devices](https://mlsanity.com/paper/2208.09577/) (2022)
- [Globally Optimized Mutual Influence Aware Ranking in E-Commerce Search](https://mlsanity.com/paper/1805.08524/) (2018)
- [Revisit Recommender System in the Permutation Prospective](https://mlsanity.com/paper/2102.12057/) (2021)
- [GRN: Generative Rerank Network for Context-wise Recommendation](https://mlsanity.com/paper/2104.00860/) (2021)
- [Image Matters: Visually modeling user behaviors using Advanced Model Server](https://mlsanity.com/paper/1711.06505/) (2017)
- [PIER: Permutation-Level Interest-Based End-to-End Re-ranking Framework in E-commerce](https://mlsanity.com/paper/2302.03487/) (2023)
- [A Versatile Framework for Evaluating Ranked Lists in terms of Group Fairness and Relevance](https://mlsanity.com/paper/2204.00280/) (2022)
- [Optimizing E-commerce Search: Toward a Generalizable and Rank-Consistent Pre-Ranking Model](https://mlsanity.com/paper/2405.05606/) (2024)
- [VLSNR:Vision-Linguistics Coordination Time Sequence-aware News Recommendation](https://mlsanity.com/paper/2210.02946/) (2022)

## Built on by (most cited first, 2 of 2)

- [UniRank: End-to-End Domain-Specific Reranking of Hybrid Text-Image Candidates](https://mlsanity.com/paper/2603.29897/) (2026)
- [MMRM: A Multiplex Multimodal Representation Model for Product Ranking in E-commerce Search](https://mlsanity.com/paper/2607.11030/) (2026)

## Abstract (arXiv)

In the rapidly evolving field of e-commerce, the effectiveness of search re-ranking models is crucial for enhancing user experience and driving conversion rates. Despite significant advancements in feature representation and model architecture, the integration of multimodal information remains underexplored. This study addresses this gap by investigating the computation and fusion of textual and visual information in the context of re-ranking. We propose Advancing Re-Ranking with Multimodal Fusion and Target-Oriented Auxiliary Tasks (ARMMT), which integrates an attention-based multimodal fusion technique and an auxiliary ranking-aligned task to enhance item representation and improve targeting capabilities. This method not only enriches the understanding of product attributes but also enables more precise and personalized recommendations. Experimental evaluations on JD.com's search platform demonstrate that ARMMT achieves state-of-the-art performance in multimodal information integration, evidenced by a 0.22% increase in the Conversion Rate (CVR), significantly contributing to Gross Merchandise Volume (GMV). This pioneering approach has the potential to revolutionize e-commerce re-ranking, leading to elevated user satisfaction and business growth.