mlsanityData to 14 Oct 2026

Captions Speak Louder than Images: Generalizing Foundation Models for E-commerce from High-quality Multimodal Instruction Data

arXiv 2410.17337 ↗First posted 22 Oct 2024cs.CL cs.AI cs.IRcited by 6

Not on the map. Why?

Abstract (arXiv)

Leveraging multimodal data to drive breakthroughs in e-commerce applications through Multimodal Foundation Models (MFMs) is gaining increasing attention from the research community. However, there are significant challenges that hinder the optimal use of multimodal e-commerce data by foundation models: (1) the scarcity of large-scale, high-quality multimodal benchmark datasets; and (2) the lack of effective multimodal information integration methods. To address these challenges, in this paper, we introduce MMECInstruct, the first-ever, large-scale, and high-quality multimodal instruction dataset for e-commerce. We also develop CASLIE, a simple, lightweight, yet effective framework for integrating multimodal information for e-commerce. Leveraging MMECInstruct, we fine-tune a series of e-commerce MFMs within CASLIE, denoted as CASLIE models. Our comprehensive evaluation demonstrates that CASLIE models substantially outperform 5 categories of advanced baseline models in the in-domain evaluation. Moreover, CASLIE models show strong generalizability to out-of-domain settings. MMECInstruct and CASLIE models are publicly accessible through https://ninglab.github.io/CASLIE/.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 37 of its 66 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

37 of 66 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Segment Anythingcited by 8,234
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
25 more
  1. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  2. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  3. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  4. 2024Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoderscited by 224
  5. 2023BioCLIP: A Vision Foundation Model for the Tree of Lifecited by 161
  6. 2023Text Is All You Need: Learning Language Representations for Sequential Recommendationcited by 149
  7. 2021Causal Intervention for Leveraging Popularity Bias in Recommendationcited by 116
  8. 2022Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Searchcited by 65
  9. 2024RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesiscited by 58
  10. 2021MAVE: A Product Dataset for Multi-source Attribute Value Extractioncited by 53
  11. 2021Deconfounded Recommendation for Alleviating Bias Amplificationcited by 51
  12. 2023EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commercecited by 49
  13. 2024What to align in multimodal contrastive learning?cited by 47
  14. 2024eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Datacited by 46
  15. 2019AmazonQA: A Review-Based Question Answering Taskcited by 43
  16. 2022Contrastive language and vision learning of general fashion conceptscited by 42
  17. 2023Amazon-M2: A Multilingual Multi-locale Shopping Session Dataset for Recommendation and Text Generationcited by 42
  18. 2020Multimodal Joint Attribute Prediction and Value Extraction for E-commerce Productcited by 39
  19. 2019Product Knowledge Graph Embedding for E-commercecited by 36
  20. 2023MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence Representation for Recommendationcited by 31
  21. 2023gSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative Samplingcited by 29
  22. 2022CommerceMM: Large-Scale Commerce MultiModal Representation Learning with Omni Retrievalcited by 25
  23. 2023Learning Instance-Level Representation for Large-Scale Multi-Modal Pretraining in E-commercecited by 8
  24. 2024A Multimodal In-Context Tuning Approach for E-Commerce Product Description Generationcited by 4
  25. 2023Multi-modality Meets Re-learning: Mitigating Negative Transfer in Sequential Recommendationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025EcomMMMU: Strategic Utilization of Visuals for Robust Multimodal E-commerce Modelscited by 5
  2. 2025MOON2.0: Dynamic Modality-balanced Multimodal Representation Learning for E-commerce Product Understandingcited by 5
  3. 2026MOON3.0: Reasoning-aware Multimodal Representation Learning for E-commerce Product Understandingcited by 4
  4. 2025Towards Reliable Evaluation of Large Language Models for Multilingual and Multimodal E-Commerce Applicationscited by 1
  5. 2026Valley3: Scaling Omni Foundation Models for E-commercecited by 1
  6. 2026Uncertainty as a Planning Signal: Multi-Turn Decision Making for Goal-Oriented Conversationcited by 0