# Is the Platform Part of the Measurement? A Protocol and Simulation Study for Cross-Platform Equivalence of Machine-Labelled Policy Sentiment

arXiv: https://arxiv.org/abs/2610.16757 · HTML: https://mlsanity.com/paper/2610.16757/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 52 references that are in the map.
- First posted on arXiv 12 Oct 2026.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Crowdsourcing a Word-Emotion Association Lexicon](https://mlsanity.com/paper/1308.6297/) (2013)
- [Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models](https://mlsanity.com/paper/2306.04746/) (2023)
- [Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts](https://mlsanity.com/paper/2407.10747/) (2024)
- [Toxic Bias: Perspective API Misreads German as More Toxic](https://mlsanity.com/paper/2312.12651/) (2023)
- [Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI](https://mlsanity.com/paper/2409.09467/) (2024)
- [Is the Sample Good Enough? Comparing Data from Twitter's Streaming API with Twitter's Firehose](https://mlsanity.com/paper/1306.5204/) (2013)
- [Big Questions for Social Media Big Data: Representativeness, Validity and Other Methodological Pitfalls](https://mlsanity.com/paper/1403.7400/) (2014)
- [Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking](https://mlsanity.com/paper/2604.11581/) (2026)
- [TED-On: A Total Error Framework for Digital Traces of Human Behavior on Online Platforms](https://mlsanity.com/paper/1907.08228/) (2019)
- [Validating daily social media macroscopes of emotions](https://mlsanity.com/paper/2108.07646/) (2021)
- [Variance-Aware LLM Annotation for Strategy Research: Sources, Diagnostics, and a Protocol for Reliable Measurement](https://mlsanity.com/paper/2601.02370/) (2025)
- [The Great Data Standoff: Researchers vs. Platforms Under the Digital Services Act](https://mlsanity.com/paper/2505.01122/) (2025)
- [Cross-Platform Short-Video Diplomacy: Topic and Sentiment Analysis of China-US Relations on Douyin and TikTok](https://mlsanity.com/paper/2510.22415/) (2025)
- [DIF Statistical Inference without Knowing Anchoring Items](https://mlsanity.com/paper/2110.11112/) (2021)
- [Different Affordances on Facebook and SMS Text Messaging Do Not Impede Generalization of Language-Based Predictive Models](https://mlsanity.com/paper/2202.01802/) (2022)
- [Rating the Raters: Rasch Measurement Theory for LLM Evaluation](https://mlsanity.com/paper/2608.27463/) (2026)

## Abstract (arXiv)

Comparing machine-labelled sentiment across social media platforms is now routine: differences in classifier scores between platforms are read as differences in public opinion. These comparisons rest on an assumption that the classifier measures the same construct, on the same scale, on every platform. Some studies check a classifier's validity platform by platform, but to our knowledge none has modelled the platform as a facet of the classifier's measurement. We adapt the psychometric equivalence toolkit (generalizability theory, measurement invariance and differential item functioning, and score linking) to machine-labelled policy sentiment, treating the platform as a survey mode, and specify a six-step protocol with anchor events fixed in advance, an anchor-density audit, and a two-classifier convergence control. A simulation with known ground truth illustrates what the assumption can cost in one generating configuration fixed in advance: across 500 replications, because the platforms measure through different functions, a pooled index that takes platform scores at face value reports the wrong sign of aggregate sentiment on nearly a quarter of days, whereas an index adjusted with each platform's known measurement function does so on under one day in sixty on average. A case study of the instrument from the first author's MSc dissertation, a fine-tuned BERT classifier over 255,446 UK economic-policy comments from Twitter/X, Reddit, YouTube, and Quora, motivates the design and illustrates the instrument-dependence the protocol addresses. The full four-platform application, under fresh collection with the pre-specified design, is the designated follow-up study. The protocol lets a researcher test whether pooling platforms is defensible in a given corpus, or whether platform scores must be linked before any average is taken.