# Domain Adaptation of Foundation LLMs for e-Commerce

arXiv: https://arxiv.org/abs/2501.09706 · HTML: https://mlsanity.com/paper/2501.09706/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 33 references that are in the map.
- First posted on arXiv 16 Jan 2025.

## Builds on (most cited first, 30 of 30)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [Llemma: An Open Language Model For Mathematics](https://mlsanity.com/paper/2310.10631/) (2023)
- [MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning](https://mlsanity.com/paper/2310.16049/) (2023)
- [Continual Pre-training of Language Models](https://mlsanity.com/paper/2302.03241/) (2023)
- [SaulLM-7B: A pioneering Large Language Model for Law](https://mlsanity.com/paper/2403.03883/) (2024)
- [Continual Pre-Training of Large Language Models: How to (re)warm your model?](https://mlsanity.com/paper/2308.04014/) (2023)
- [LLaMA Pro: Progressive LLaMA with Block Expansion](https://mlsanity.com/paper/2401.02415/) (2024)
- [WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models](https://mlsanity.com/paper/2112.06598/) (2021)
- [Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models](https://mlsanity.com/paper/2407.07263/) (2024)
- [USP: A Unified Sequence Parallelism Approach for Long Context Generative AI](https://mlsanity.com/paper/2405.07719/) (2024)
- [EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commerce](https://mlsanity.com/paper/2308.06966/) (2023)
- [eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data](https://mlsanity.com/paper/2402.08831/) (2024)
- [SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain](https://mlsanity.com/paper/2407.19584/) (2024)
- [A Continued Pretrained LLM Approach for Automatic Medical Note Generation](https://mlsanity.com/paper/2403.09057/) (2024)
- [Demystifying Domain-adaptive Post-training for Financial LLMs](https://mlsanity.com/paper/2501.04961/) (2025)
- [LiLiuM: eBay's Large Language Models for e-commerce](https://mlsanity.com/paper/2406.12023/) (2024)

## Built on by (most cited first, 9 of 9)

- [ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistants](https://mlsanity.com/paper/2601.18225/) (2026)
- [Genesis: Evolving Attack Strategies for LLM Web Agent Red-Teaming](https://mlsanity.com/paper/2510.18314/) (2025)
- [Unilogit: Robust Machine Unlearning for LLMs Using Uniform-Target Self-Distillation](https://mlsanity.com/paper/2505.06027/) (2025)
- [Vocabulary Customization for Efficient Domain-Specific LLM Deployment](https://mlsanity.com/paper/2509.26124/) (2025)
- [Adapting Vision-Language Models for E-commerce Understanding at Scale](https://mlsanity.com/paper/2602.11733/) (2026)
- [Valley3: Scaling Omni Foundation Models for E-commerce](https://mlsanity.com/paper/2605.01278/) (2026)
- [JD Oxygen AI Item Center (Oxygen AIIC) V1: An Industrial-Scale LLM/VLM-Centric Solution for Item Understanding, Management, and Applications](https://mlsanity.com/paper/2606.28070/) (2026)
- [Beyond Relevance: Structured Semantic Supervision for Product Search with LLM-Augmented Annotations](https://mlsanity.com/paper/2609.23646/) (2026)
- [From a Large-Scale Teacher to Efficient Multimodal Students for Search Relevance Prediction](https://mlsanity.com/paper/2610.16569/) (2026)

## Abstract (arXiv)

We present the e-Llama models: 8 billion and 70 billion parameter large language models that are adapted towards the e-commerce domain. These models are meant as foundation models with deep knowledge about e-commerce, that form a base for instruction- and fine-tuning. The e-Llama models are obtained by continuously pretraining the Llama 3.1 base models on 1 trillion tokens of domain-specific data. We discuss our approach and motivate our choice of hyperparameters with a series of ablation studies. To quantify how well the models have been adapted to the e-commerce domain, we define and implement a set of multilingual, e-commerce specific evaluation tasks. We show that, when carefully choosing the training setup, the Llama 3.1 models can be adapted towards the new domain without sacrificing significant performance on general domain tasks. We also explore the possibility of merging the adapted model and the base model for a better control of the performance trade-off between domains.