mlsanityData to 14 Oct 2026

LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset

arXiv 2309.11998 ↗First posted 21 Sep 2023cs.CL cs.AIcited by 461

Not on the map. Why?

Cited by 57 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Studying how people interact with large language models (LLMs) in real-world scenarios is increasingly important due to their widespread use in various applications. In this paper, we introduce LMSYS-Chat-1M, a large-scale dataset containing one million real-world conversations with 25 state-of-the-art LLMs. This dataset is collected from 210K unique IP addresses in the wild on our Vicuna demo and Chatbot Arena website. We offer an overview of the dataset's content, including its curation process, basic statistics, and topic distribution, highlighting its diversity, originality, and scale. We demonstrate its versatility through four use cases: developing content moderation models that perform similarly to GPT-4, building a safety benchmark, training instruction-following models that perform similarly to Vicuna, and creating challenging benchmark questions. We believe that this dataset will serve as a valuable resource for understanding and advancing LLM capabilities. The dataset is publicly available at https://huggingface.co/datasets/lmsys/lmsys-chat-1m.

Key facts. Cited by 461 papers in the map (data through 14 Oct 2026). Builds on 36 of its 64 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

36 of 64 references.

References · 36

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  11. 2020Extracting Training Data from Large Language Modelscited by 2,189
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,139
24 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  2. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  4. 2023PaLM 2 Technical Reportcited by 1,162
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  7. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  9. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  10. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
  11. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  12. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  13. 2018CoQA: A Conversational Question Answering Challengecited by 683
  14. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
  15. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  16. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  17. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
  18. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  19. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  20. 2015The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systemscited by 172
  21. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 168
  22. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 91
  23. 2023INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Modelscited by 62
  24. 2023On Optimal Caching and Model Multiplexing for Large Model Inferencecited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 102; 2025: 175; 2026: 170.

461 papers.

Sort:

By area:

  1. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  2. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  4. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  5. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  6. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  7. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  8. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  9. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
  10. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  11. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  12. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
38 more
  1. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  2. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  3. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 148
  4. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 144
  5. 2024Nemotron-4 340B Technical Reportcited by 144
  6. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 130
  7. 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 129
  8. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  9. 2023Online Speculative Decodingcited by 113
  10. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  11. 2023Generative Language Models Exhibit Social Identity Biasescited by 111
  12. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  13. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  14. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 95
  15. 2024Watermark Stealing in Large Language Modelscited by 92
  16. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 90
  17. 2025Persona Features Control Emergent Misalignmentcited by 86
  18. 2024Efficient LLM Scheduling by Learning to Rankcited by 85
  19. 2023Fairness in Serving Large Language Modelscited by 78
  20. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 77
  21. 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
  22. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 75
  23. 2024Clio: Privacy-Preserving Insights into Real-World AI Usecited by 72
  24. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 71
  25. 2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computationcited by 70
  26. 2024SafeBench: A Safety Evaluation Framework for Multimodal Large Language Modelscited by 69
  27. 2024Instruction Tuning for Secure Code Generationcited by 68
  28. 2025STAR-1: Safer Alignment of Reasoning LLMs with 1K Datacited by 68
  29. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 66
  30. 2024Exploiting LLM Quantizationcited by 65
  31. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 63
  32. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 59
  33. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 59
  34. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 58
  35. 2025Flipping the Dialogue: Training and Evaluating User Language Modelscited by 58
  36. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 57
  37. 2024AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learningcited by 53
  38. 2025The Best Instruction-Tuning Data are Those That Fitcited by 52

50 most cited of 461 All 461 (JSON)