mlsanityData to 14 Oct 2026

Improving Your Model Ranking on Chatbot Arena by Vote Rigging

arXiv 2501.17858 ↗First posted 29 Jan 2025cs.CL cs.AI cs.CRcited by 17

Not on the map. Why?

Abstract (arXiv)

Chatbot Arena is a popular platform for evaluating LLMs by pairwise battles, where users vote for their preferred response from two randomly sampled anonymous models. While Chatbot Arena is widely regarded as a reliable LLM ranking leaderboard, we show that crowdsourced voting can be rigged to improve (or decrease) the ranking of a target model mt. We first introduce a straightforward target-only rigging strategy that focuses on new battles involving mt, identifying it via watermarking or a binary classifier, and exclusively voting for mt wins. However, this strategy is practically inefficient because there are over 190 models on Chatbot Arena and on average only about 1% of new battles will involve mt. To overcome this, we propose omnipresent rigging strategies, exploiting the Elo rating mechanism of Chatbot Arena that any new vote on a battle can influence the ranking of the target model mt, even if mt is not directly involved in the battle. We conduct experiments on around 1.7 million historical votes from the Chatbot Arena Notebook, showing that omnipresent rigging strategies can improve model rankings by rigging only hundreds of new votes. While we have evaluated several defense mechanisms, our findings highlight the importance of continued efforts to prevent vote rigging. Our code is available at https://github.com/sail-sg/Rigging-ChatbotArena.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 49 of its 69 references that are in the map. First posted on arXiv 29 Jan 2025.

Builds on

49 of 69 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  12. 2023Qwen Technical Reportcited by 3,344
37 more
  1. 2023Mistral 7Bcited by 3,164
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  5. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  7. 2023A Watermark for Large Language Modelscited by 725
  8. 2024Yi: Open Foundation Models by 01.AIcited by 719
  9. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  10. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  11. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 481
  12. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  13. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  14. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  15. 2023Undetectable Watermarks for Language Modelscited by 226
  16. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 195
  17. 2023On the Reliability of Watermarks for Large Language Modelscited by 189
  18. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  19. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  20. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 157
  21. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 143
  22. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 128
  23. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  24. 2023On the Exploitability of Instruction Tuningcited by 116
  25. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 110
  26. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 97
  27. 2024BadEdit: Backdooring large language models by model editingcited by 96
  28. 2023Ghostbuster: Detecting Text Ghostwritten by Large Language Modelscited by 91
  29. 2021Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Textcited by 89
  30. 2023Three Bricks to Consolidate Watermarks for Large Language Modelscited by 84
  31. 2023Robust Multi-bit Natural Language Watermarking through Invariant Featurescited by 82
  32. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 45
  33. 2023GPT-Sentinel: Distinguishing Human and ChatGPT Generated Contentcited by 43
  34. 2023Token Prediction as Implicit Classification to Identify LLM-Generated Textcited by 27
  35. 2025Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboardscited by 20
  36. 2024Challenges in Trustworthy Human Evaluation of Chatbotscited by 14
  37. 2021Guiding Neural Story Generation with Reader Modelscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 7.

17 papers.

Sort:

By area:

  1. 2025PointArena: Probing Multimodal Grounding Through Language-Guided Pointingcited by 31
  2. 2025EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Treescited by 27
  3. 2025Ranked Voting based Self-Consistency of Large Language Modelscited by 19
  4. 2025Life-Cycle Routing Vulnerabilities of LLM Routercited by 11
  5. 2025Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankingscited by 8
  6. 2025Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoptioncited by 7
  7. 2025Exploiting Leaderboards for Large-Scale Distribution of Malicious Modelscited by 5
  8. 2026Who Defines "Best"? Towards Interactive, User-Defined Evaluation of LLM Leaderboardscited by 2
  9. 2026A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulationcited by 2
  10. 2026Strategic Candidacy in Generative AI Arenascited by 1
  11. 2025The Generative Energy Arena (GEA): Incorporating Energy Awareness in Large Language Model (LLM) Human Evaluationscited by 0
  12. 2025Inclusion Arena: An Open Platform for Evaluating Large Foundation Models with Real-World Appscited by 0
5 more
  1. 2025Text-to-Image Models Leave Identifiable Signatures: Implications for Leaderboard Securitycited by 0
  2. 2026Identifying Models Behind Text-to-Image Leaderboardscited by 0
  3. 2026InterPol: De-anonymizing LM Arena via Interpolated Preference Learningcited by 0
  4. 2026Why Global LLM Leaderboards Are Misleading: Small Portfolios for Heterogeneous Supervised MLcited by 0
  5. 2026How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Prompt-to-Leaderboard
  2. 2024Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arena
  3. 2024Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions
  4. 2026Low Rank for Rank: Uncertainty-Aware Task-Specific LLM Ranking under Sparse Pairwise Comparisons
  5. 2023Safeguarding Crowdsourcing Surveys from ChatGPT with Prompt Injection
  6. 2026Personalized Benchmarking: Evaluating LLMs by Individual Preferences
  7. 2025Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information
  8. 2025CHARM: Calibrating Reward Models With Chatbot Arena Scores