# LLM-Ensemble: Optimal Large Language Model Ensemble Method for E-commerce Product Attribute Value Extraction

arXiv: https://arxiv.org/abs/2403.00863 · HTML: https://mlsanity.com/paper/2403.00863/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 36 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Code4Struct: Code Generation for Few-Shot Event Structure Prediction](https://mlsanity.com/paper/2210.12810/) (2022)
- [OpenTag: Open Attribute Value Extraction from Product Profiles [Deep Learning, Active Learning, Named Entity Recognition]](https://mlsanity.com/paper/1806.01264/) (2018)
- [MAVE: A Product Dataset for Multi-source Attribute Value Extraction](https://mlsanity.com/paper/2112.08663/) (2021)
- [AdaTag: Multi-Attribute Value Extraction from Product Profiles with Adaptive Decoding](https://mlsanity.com/paper/2106.02318/) (2021)
- [TXtract: Taxonomy-Aware Knowledge Extraction for Thousands of Product Categories](https://mlsanity.com/paper/2004.13852/) (2020)
- [GENEVA: Benchmarking Generalizability for Event Argument Extraction with Hundreds of Event Types and Argument Roles](https://mlsanity.com/paper/2205.12505/) (2022)
- [Error Rate Bounds and Iterative Weighted Majority Voting for Crowdsourcing](https://mlsanity.com/paper/1411.4086/) (2014)
- [OA-Mine: Open-World Attribute Mining for E-Commerce Products with Weak Supervision](https://mlsanity.com/paper/2204.13874/) (2022)

## Built on by (most cited first, 30 of 39)

- [A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models](https://mlsanity.com/paper/2503.23350/) (2025)
- [Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information](https://mlsanity.com/paper/2510.01499/) (2025)
- [Using LLMs for the Extraction and Normalization of Product Attribute Values](https://mlsanity.com/paper/2403.02130/) (2024)
- [ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction](https://mlsanity.com/paper/2404.15592/) (2024)
- [Bayesian Calibration of Win Rate Estimation with LLM Evaluators](https://mlsanity.com/paper/2411.04424/) (2024)
- [LLM DNA: Tracing Model Evolution via Functional Representations](https://mlsanity.com/paper/2509.24496/) (2025)
- [InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusion](https://mlsanity.com/paper/2505.13893/) (2025)
- [A Multi-LLM Debiasing Framework](https://mlsanity.com/paper/2409.13884/) (2024)
- [Evaluating and Mitigating Social Bias for Large Language Models in Open-ended Settings](https://mlsanity.com/paper/2412.06134/) (2024)
- [Prompt Optimizer of Text-to-Image Diffusion Models for Abstract Concept Understanding](https://mlsanity.com/paper/2404.11589/) (2024)
- [Triple Modality Fusion: Aligning Visual, Textual, and Graph Data with Large Language Models for Multi-Behavior Recommendations](https://mlsanity.com/paper/2410.12228/) (2024)
- [Ensemble Learning for Large Language Models in Text and Code Generation: A Survey](https://mlsanity.com/paper/2503.13505/) (2025)
- [LLMExplainer: Large Language Model based Bayesian Inference for Graph Explanation Generation](https://mlsanity.com/paper/2407.15351/) (2024)
- [Multi-LLM Collaborative Caption Generation in Scientific Documents](https://mlsanity.com/paper/2501.02552/) (2025)
- [EAVE: Efficient Product Attribute Value Extraction via Lightweight Sparse-layer Interaction](https://mlsanity.com/paper/2406.06839/) (2024)
- [DNB-AI-Project at SemEval-2025 Task 5: An LLM-Ensemble Approach for Automated Subject Indexing](https://mlsanity.com/paper/2504.21589/) (2025)
- [Agentic AI for autonomous anomaly management in complex systems](https://mlsanity.com/paper/2507.15676/) (2025)
- [Maximizing User Experience with LLMOps-Driven Personalized Recommendation Systems](https://mlsanity.com/paper/2404.00903/) (2024)
- [Exploring Large Language Models for Product Attribute Value Identification](https://mlsanity.com/paper/2409.12695/) (2024)
- [Visual Zero-Shot E-Commerce Product Attribute Value Extraction](https://mlsanity.com/paper/2502.15979/) (2025)
- [A Cost-Effective LLM-based Approach to Identify Wildlife Trafficking in Online Marketplaces](https://mlsanity.com/paper/2504.21211/) (2025)
- [Crafting Efficient Fine-Tuning Strategies for Large Language Models](https://mlsanity.com/paper/2407.13906/) (2024)
- [Self-Explained Keywords Empower Large Language Models for Code Generation](https://mlsanity.com/paper/2410.15966/) (2024)
- [Learning Personalized Prompts for Healthcare Guidance](https://mlsanity.com/paper/2412.15957/) (2024)
- [Self-Refinement Strategies for LLM-based Product Attribute Value Extraction](https://mlsanity.com/paper/2501.01237/) (2025)
- [Aug2Search: Enhancing Facebook Marketplace Search with LLM-Generated Synthetic Data Augmentation](https://mlsanity.com/paper/2505.16065/) (2025)
- [VideoAVE: A Multi-Attribute Video-to-Text Attribute Value Extraction Dataset and Benchmark Models](https://mlsanity.com/paper/2508.11801/) (2025)
- [Towards Reliable Evaluation of Large Language Models for Multilingual and Multimodal E-Commerce Applications](https://mlsanity.com/paper/2510.20632/) (2025)
- [MiRAGE: Misconception Detection with Retrieval-Guided Multi-Stage Reasoning and Ensemble Fusion](https://mlsanity.com/paper/2511.01182/) (2025)
- [Multi-LLM Query Optimization](https://mlsanity.com/paper/2603.24617/) (2026)

## Abstract (arXiv)

Product attribute value extraction is a pivotal component in Natural Language Processing (NLP) and the contemporary e-commerce industry. The provision of precise product attribute values is fundamental in ensuring high-quality recommendations and enhancing customer satisfaction. The recently emerging Large Language Models (LLMs) have demonstrated state-of-the-art performance in numerous attribute extraction tasks, without the need for domain-specific training data. Nevertheless, varying strengths and weaknesses are exhibited by different LLMs due to the diversity in data, architectures, and hyperparameters. This variation makes them complementary to each other, with no single LLM dominating all others. Considering the diverse strengths and weaknesses of LLMs, it becomes necessary to develop an ensemble method that leverages their complementary potentials. In this paper, we propose a novel algorithm called LLM-ensemble to ensemble different LLMs' outputs for attribute value extraction. We iteratively learn the weights for different LLMs to aggregate the labels with weights to predict the final attribute value. Not only can our proposed method be proven theoretically optimal, but it also ensures efficient computation, fast convergence, and safe deployment. We have also conducted extensive experiments with various state-of-the-art LLMs, including Llama2-13B, Llama2-70B, PaLM-2, GPT-3.5, and GPT-4, on Walmart's internal data. Our offline metrics demonstrate that the LLM-ensemble method outperforms all the state-of-the-art single LLMs on Walmart's internal dataset. This method has been launched in several production models, leading to improved Gross Merchandise Volume (GMV), Click-Through Rate (CTR), Conversion Rate (CVR), and Add-to-Cart Rate (ATC).