# Fine-tuning Multimodal Large Language Models for Product Bundling

arXiv: https://arxiv.org/abs/2407.11712 · HTML: https://mlsanity.com/paper/2407.11712/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 33 of its 63 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks](https://mlsanity.com/paper/2305.11175/) (2023)
- [TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation](https://mlsanity.com/paper/2305.00447/) (2023)
- [Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1802.05814/) (2018)
- [Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration](https://mlsanity.com/paper/2306.09093/) (2023)
- [From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models](https://mlsanity.com/paper/2212.10846/) (2022)
- [ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions](https://mlsanity.com/paper/2303.06594/) (2023)
- [Can GNN be Good Adapter for LLMs?](https://mlsanity.com/paper/2402.12984/) (2024)
- [Learning Fashion Compatibility with Bidirectional LSTMs](https://mlsanity.com/paper/1707.05691/) (2017)
- [POG: Personalized Outfit Generation for Fashion Recommendation at Alibaba iFashion](https://mlsanity.com/paper/1905.01866/) (2019)
- [GraphTranslator: Aligning Graph Model to Large Language Model for Open-ended Tasks](https://mlsanity.com/paper/2402.07197/) (2024)
- [SpeechGPT-Gen: Scaling Chain-of-Information Speech Generation](https://mlsanity.com/paper/2401.13527/) (2024)
- [Hierarchical Fashion Graph Network for Personalized Outfit Recommendation](https://mlsanity.com/paper/2005.12566/) (2020)
- [CrossCBR: Cross-view Contrastive Learning for Bundle Recommendation](https://mlsanity.com/paper/2206.00242/) (2022)
- [Prompt Learning for News Recommendation](https://mlsanity.com/paper/2304.05263/) (2023)
- [Personalized Bundle List Recommendation](https://mlsanity.com/paper/1904.01933/) (2019)
- [Personalized Bundle Recommendation in Online Games](https://mlsanity.com/paper/2104.05307/) (2021)
- [Leveraging Multimodal Features and Item-level User Feedback for Bundle Construction](https://mlsanity.com/paper/2310.18770/) (2023)
- [An Analysis of Approaches Taken in the ACM RecSys Challenge 2018 for Automatic Music Playlist Continuation](https://mlsanity.com/paper/1810.01520/) (2018)
- [Towards Personalized Bundle Creative Generation with Contrastive Non-Autoregressive Decoding](https://mlsanity.com/paper/2205.14970/) (2022)
- [Consistency-Aware Recommendation for User-Generated ItemList Continuation](https://mlsanity.com/paper/1912.13031/) (2019)
- [Improving Outfit Recommendation with Co-supervision of Fashion Generation](https://mlsanity.com/paper/1908.09104/) (2019)

## Built on by (most cited first, 9 of 9)

- [L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models](https://mlsanity.com/paper/2505.17505/) (2025)
- [Preference Diffusion for Recommendation](https://mlsanity.com/paper/2410.13117/) (2024)
- [Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfer](https://mlsanity.com/paper/2602.19058/) (2026)
- [Distilling Transitional Pattern to Large Language Models for Multimodal Session-based Recommendation](https://mlsanity.com/paper/2504.10538/) (2025)
- [Does Knowledge Distillation Matter for Large Language Model based Bundle Generation?](https://mlsanity.com/paper/2504.17220/) (2025)
- [Routing Distilled Knowledge via Mixture of LoRA Experts for Large Language Model based Bundle Generation](https://mlsanity.com/paper/2508.17250/) (2025)
- [An item is worth one token in Multimodal Large Language Models-based Sequential Recommendation](https://mlsanity.com/paper/2511.05885/) (2025)
- [FashionStylist: An Expert Knowledge-enhanced Multimodal Dataset for Fashion Understanding](https://mlsanity.com/paper/2604.09249/) (2026)
- [Dual-Enhancement Product Bundling: Bridging Interactive Graph and Large Language Model](https://mlsanity.com/paper/2604.14030/) (2026)

## Abstract (arXiv)

Recent advances in product bundling have leveraged multimodal information through sophisticated encoders, but remain constrained by limited semantic understanding and a narrow scope of knowledge. Therefore, some attempts employ In-context Learning (ICL) to explore the potential of large language models (LLMs) for their extensive knowledge and complex reasoning abilities. However, these efforts are inadequate in understanding mulitmodal data and exploiting LLMs' knowledge for product bundling. To bridge the gap, we introduce Bundle-MLLM, a novel framework that fine-tunes LLMs through a hybrid item tokenization approach within a well-designed optimization strategy. Specifically, we integrate textual, media, and relational data into a unified tokenization, introducing a soft separation token to distinguish between textual and non-textual tokens. Additionally, a streamlined yet powerful multimodal fusion module is employed to embed all non-textual features into a single, informative token, significantly boosting efficiency. To tailor product bundling tasks for LLMs, we reformulate the task as a multiple-choice question with candidate items as options. We further propose a progressive optimization strategy that fine-tunes LLMs for disentangled objectives: 1) learning bundle patterns and 2) enhancing multimodal semantic understanding specific to product bundling. Extensive experiments on four datasets across two domains demonstrate that our approach outperforms a range of state-of-the-art (SOTA) methods.