mlsanityData to 7 Oct 2026

Challenge design roadmap

arXiv 2401.13693 ↗First posted 15 Jan 2024cs.OH cs.AI cs.HCcited by 0

Not on the map. Why?

Abstract (arXiv)

Challenges can be seen as a type of game that motivates participants to solve serious tasks. As a result, competition organizers must develop effective game rules. However, these rules have multiple objectives beyond making the game enjoyable for participants. These objectives may include solving real-world problems, advancing scientific or technical areas, making scientific discoveries, and educating the public. In many ways, creating a challenge is similar to launching a product. It requires the same level of excitement and rigorous testing, and the goal is to attract ''customers'' in the form of participants. The process begins with a solid plan, such as a competition proposal that will eventually be submitted to an international conference and subjected to peer review. Although peer review does not guarantee quality, it does force organizers to consider the impact of their challenge, identify potential oversights, and generally improve its quality. This chapter provides guidelines for creating a strong plan for a challenge. The material draws on the preparation guidelines from organizations such as Kaggle 1 , ChaLearn 2 and Tailor 3 , as well as the NeurIPS proposal template, which some of the authors contributed to.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 42 references that are in the map. First posted on arXiv 15 Jan 2024.

Builds on

17 of 42 references.

References · 17

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  3. 2018Datasheets for Datasetscited by 1,371
  4. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  5. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  6. 2014OpenML: networked science in machine learningcited by 523
  7. 2018Meta-Transfer Learning for Few-Shot Learningcited by 306
  8. 2020Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformationcited by 183
  9. 2016Learning a Deep Embedding Model for Zero-Shot Learningcited by 153
  10. 2018Task-Agnostic Meta-Learning for Few-shot Learningcited by 151
  11. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 100
  12. 2020Self-training for Few-shot Transfer Across Extreme Task Differencescited by 75
5 more
  1. 2015The Ladder: A Reliable Leaderboard for Machine Learning Competitionscited by 67
  2. 2021Learning to run a Power Network Challenge: a Retrospective Analysiscited by 52
  3. 2019Learning to run a power network challenge for training topology controllerscited by 33
  4. 2022Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019cited by 16
  5. 2022Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classificationcited by 10

Built on by

Not cited yet.