# Making Data Work Count

arXiv: https://arxiv.org/abs/2311.18046 · HTML: https://mlsanity.com/paper/2311.18046/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 90 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Builds on (most cited first, 11 of 11)

- [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020)
- [Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure](https://mlsanity.com/paper/2010.13561/) (2020)
- [Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem](https://mlsanity.com/paper/2310.02521/) (2023)
- [Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practices](https://mlsanity.com/paper/2006.12358/) (2020)
- [Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision](https://mlsanity.com/paper/2007.14886/) (2020)
- [Whose AI Dream? In search of the aspiration in data annotation](https://mlsanity.com/paper/2203.10748/) (2022)
- [Quantifying the Invisible Labor in Crowd Work](https://mlsanity.com/paper/2110.00169/) (2021)
- [The Data-Production Dispositif](https://mlsanity.com/paper/2205.11963/) (2022)
- [Studying Up Machine Learning Data: Why Talk About Bias When We Mean Power?](https://mlsanity.com/paper/2109.08131/) (2021)
- [Skin Deep: Investigating Subjectivity in Skin Tone Annotations for Computer Vision Benchmark Datasets](https://mlsanity.com/paper/2305.09072/) (2023)
- [Human-Centered Responsible Artificial Intelligence: Current & Future Trends](https://mlsanity.com/paper/2302.08157/) (2023)

## Built on by (most cited first, 2 of 2)

- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [How Hyper-Datafication Impacts the Sustainability Costs in Frontier AI](https://mlsanity.com/paper/2602.00056/) (2026)

## Abstract (arXiv)

In this paper, we examine the work of data annotation. Specifically, we focus on the role of counting or quantification in organising annotation work. Based on an ethnographic study of data annotation in two outsourcing centres in India, we observe that counting practices and its associated logics are an integral part of day-to-day annotation activities. In particular, we call attention to the presumption of total countability observed in annotation - the notion that everything, from tasks, datasets and deliverables, to workers, work time, quality and performance, can be managed by applying the logics of counting. To examine this, we draw on sociological and socio-technical scholarship on quantification and develop the lens of a 'regime of counting' that makes explicit the specific counts, practices, actors and structures that underpin the pervasive counting in annotation. We find that within the AI supply chain and data work, counting regimes aid the assertion of authority by the AI clients (also called requesters) over annotation processes, constituting them as reductive, standardised, and homogenous. We illustrate how this has implications for i) how annotation work and workers get valued, ii) the role human discretion plays in annotation, and iii) broader efforts to introduce accountable and more just practices in AI. Through these implications, we illustrate the limits of operating within the logic of total countability. Instead, we argue for a view of counting as partial - located in distinct geographies, shaped by specific interests and accountable in only limited ways. This, we propose, sets the stage for a fundamentally different orientation to counting and what counts in data annotation.