mlsanityData to 14 Oct 2026

AI Competitions and Benchmarks: Dataset Development

arXiv 2404.09703 ↗First posted 15 Apr 2024cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Machine learning is now used in many applications thanks to its ability to predict, generate, or discover patterns from large quantities of data. However, the process of collecting and transforming data for practical use is intricate. Even in today's digital era, where substantial data is generated daily, it is uncommon for it to be readily usable; most often, it necessitates meticulous manual data preparation. The haste in developing new models can frequently result in various shortcomings, potentially posing risks when deployed in real-world scenarios (eg social discrimination, critical failures), leading to the failure or substantial escalation of costs in AI-based projects. This chapter provides a comprehensive overview of established methodological tools, enriched by our practical experience, in the development of datasets for machine learning. Initially, we develop the tasks involved in dataset development and offer insights into their effective management (including requirements, design, implementation, evaluation, distribution, and maintenance). Then, we provide more details about the implementation process which includes data collection, transformation, and quality evaluation. Finally, we address practical considerations regarding dataset distribution and maintenance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 54 of its 136 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

54 of 136 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  6. 2022Generative Adversarial Networkscited by 5,627
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  12. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
38 more
  1. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  2. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  3. 2018Datasheets for Datasetscited by 1,386
  4. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  5. 2017Counterfactual Fairnesscited by 804
  6. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
  7. 2014OpenML: networked science in machine learningcited by 526
  8. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 372
  9. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  10. 2018AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Biascited by 319
  11. 2017Grammar Variational Autoencodercited by 285
  12. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 274
  13. 2017graph2vec: Learning Distributed Representations of Graphscited by 230
  14. 2021Fake It Till You Make It: Face analysis in the wild using synthetic data alonecited by 200
  15. 2016Imbalanced-learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learningcited by 195
  16. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  17. 2017OpenML Benchmarking Suitescited by 178
  18. 2020Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructurecited by 138
  19. 2020REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasetscited by 136
  20. 2013Generalized Denoising Auto-Encoders as Generative Modelscited by 113
  21. 2019Synthetic Data for Deep Learningcited by 113
  22. 2018A Survey on Data Collection for Machine Learning: a Big Data -- AI Integration Perspectivecited by 112
  23. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 102
  24. 2018The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standardscited by 97
  25. 2021Weakly Supervised Object Localization and Detection: A Surveycited by 84
  26. 2020Model Patching: Closing the Subgroup Performance Gap with Data Augmentationcited by 83
  27. 2020Face Recognition: Too Bias, or Not Too Bias?cited by 82
  28. 2020Radioactive data: tracing through trainingcited by 67
  29. 2022The Effects of Regularization and Data Augmentation are Class Dependentcited by 67
  30. 2016Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applicationscited by 45
  31. 2023Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classificationcited by 30
  32. 2022Learning Curves for Decision Making in Supervised Machine Learning: A Surveycited by 27
  33. 2018Feature Space Transfer for Data Augmentationcited by 21
  34. 2019Towards Standardization of Data Licenses: The Montreal Data Licensecited by 19
  35. 2022Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Taskscited by 17
  36. 2021Fast and Informative Model Selection using Learning Curve Cross-Validationcited by 15
  37. 2022DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systemscited by 14
  38. 2019On the Effect of Observed Subject Biases in Apparent Personality Analysis from Audio-visual Signalscited by 11

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.