mlsanityData to 7 Oct 2026

Domain Adaptation of Foundation LLMs for e-Commerce

arXiv 2501.09706 ↗First posted 16 Jan 2025cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

We present the e-Llama models: 8 billion and 70 billion parameter large language models that are adapted towards the e-commerce domain. These models are meant as foundation models with deep knowledge about e-commerce, that form a base for instruction- and fine-tuning. The e-Llama models are obtained by continuously pretraining the Llama 3.1 base models on 1 trillion tokens of domain-specific data. We discuss our approach and motivate our choice of hyperparameters with a series of ablation studies. To quantify how well the models have been adapted to the e-commerce domain, we define and implement a set of multilingual, e-commerce specific evaluation tasks. We show that, when carefully choosing the training setup, the Llama 3.1 models can be adapted towards the new domain without sacrificing significant performance on general domain tasks. We also explore the possibility of merging the adapted model and the base model for a better control of the performance trade-off between domains.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 30 of its 33 references that are in the map. First posted on arXiv 16 Jan 2025.

Builds on

30 of 33 references.

References · 30

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  10. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
18 more
  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  2. 2019Publicly Available Clinical BERT Embeddingscited by 995
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  4. 2023Llemma: An Open Language Model For Mathematicscited by 385
  5. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 195
  6. 2023Continual Pre-training of Language Modelscited by 194
  7. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 136
  8. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  9. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 99
  10. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  11. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 69
  12. 2024USP: A Unified Sequence Parallelism Approach for Long Context Generative AIcited by 58
  13. 2023EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commercecited by 48
  14. 2024eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Datacited by 44
  15. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
  16. 2024A Continued Pretrained LLM Approach for Automatic Medical Note Generationcited by 30
  17. 2025Demystifying Domain-adaptive Post-training for Financial LLMscited by 24
  18. 2024LiLiuM: eBay's Large Language Models for e-commercecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2026ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistantscited by 9
  2. 2025Genesis: Evolving Attack Strategies for LLM Web Agent Red-Teamingcited by 5
  3. 2025Unilogit: Robust Machine Unlearning for LLMs Using Uniform-Target Self-Distillationcited by 4
  4. 2025Vocabulary Customization for Efficient Domain-Specific LLM Deploymentcited by 1
  5. 2026Valley3: Scaling Omni Foundation Models for E-commercecited by 1
  6. 2026Adapting Vision-Language Models for E-commerce Understanding at Scalecited by 0
  7. 2026JD Oxygen AI Item Center (Oxygen AIIC) V1: An Industrial-Scale LLM/VLM-Centric Solution for Item Understanding, Management, and Applicationscited by 0
  8. 2026Beyond Relevance: Structured Semantic Supervision for Product Search with LLM-Augmented Annotationscited by 0