mlsanityData to 7 Oct 2026

What should I wear to a party in a Greek taverna? Evaluation for Conversational Agents in the Fashion Domain

arXiv 2408.08907 ↗First posted 13 Aug 2024cs.IR cs.AI cs.CLcited by 2

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language models (LLMs) are poised to revolutionize the domain of online fashion retail, enhancing customer experience and discovery of fashion online. LLM-powered conversational agents introduce a new way of discovery by directly interacting with customers, enabling them to express in their own ways, refine their needs, obtain fashion and shopping advice that is relevant to their taste and intent. For many tasks in e-commerce, such as finding a specific product, conversational agents need to convert their interactions with a customer to a specific call to different backend systems, e.g., a search system to showcase a relevant set of products. Therefore, evaluating the capabilities of LLMs to perform those tasks related to calling other services is vital. However, those evaluations are generally complex, due to the lack of relevant and high quality datasets, and do not align seamlessly with business needs, amongst others. To this end, we created a multilingual evaluation dataset of 4k conversations between customers and a fashion assistant in a large e-commerce fashion platform to measure the capabilities of LLMs to serve as an assistant between customers and a backend engine. We evaluate a range of models, showcasing how our dataset scales to business needs and facilitates iterative development of tools.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 18 of its 46 references that are in the map. First posted on arXiv 13 Aug 2024.

Builds on

18 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk10 Jan

Other references · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  7. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  8. 2020Evaluation of Text Generation: A Surveycited by 275
  9. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  10. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  11. 2018Supervising strong learners by amplifying weak expertscited by 139
  12. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
5 more
  1. 2023Human Feedback is not Gold Standardcited by 82
  2. 2023Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactionscited by 70
  3. 2020Zero-Shot Transfer Learning with Synthesized Data for Multi-Domain Dialogue State Trackingcited by 65
  4. 2024Dialect prejudice predicts AI decisions about people's character, employability, and criminalitycited by 35
  5. 2021GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generationcited by 20

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  2. 2025Towards Conversational Development Environments: Using Theory-of-Mind and Multi-Agent Architectures for Requirements Refinementcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025FashionM3: Multimodal, Multitask, and Multiround Fashion Assistant based on Unified Vision-Language Model
  2. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey
  3. 2025Integrating Domain Knowledge into Large Language Models for Enhanced Fashion Recommendations
  4. 2025Cite Before You Speak: Enhancing Context-Response Grounding in E-commerce Conversational LLM-Agents
  5. 2026Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking
  6. 2026ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistants
  7. 2026EComStage: Stage-wise and Orientation-specific Benchmarking for Large Language Models in E-commerce
  8. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate