# Navigating the Post-API Dilemma | Search Engine Results Pages Present a Biased View of Social Media Data

arXiv: https://arxiv.org/abs/2401.15479 · HTML: https://mlsanity.com/paper/2401.15479/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 46 references that are in the map.
- First posted on arXiv 27 Jan 2024.

## Builds on (most cited first, 12 of 12)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [TimeLMs: Diachronic Language Models from Twitter](https://mlsanity.com/paper/2202.03829/) (2022)
- [Uncovering Coordinated Networks on Social Media: Methods and Case Studies](https://mlsanity.com/paper/2001.05658/) (2020)
- [Public sentiment analysis and topic modeling regarding COVID-19 vaccines on the Reddit social media platform: A call to action for strengthening vaccine confidence](https://mlsanity.com/paper/2108.13293/) (2021)
- [Is the Sample Good Enough? Comparing Data from Twitter's Streaming API with Twitter's Firehose](https://mlsanity.com/paper/1306.5204/) (2013)
- [Just Another Day on Twitter: A Complete 24 Hours of Twitter Data](https://mlsanity.com/paper/2301.11429/) (2023)
- [Towards Automated Sexual Violence Report Tracking](https://mlsanity.com/paper/1911.06961/) (2019)
- [Similarity of symbol frequency distributions with heavy tails](https://mlsanity.com/paper/1510.00277/) (2015)
- [Consumers and Curators: Browsing and Voting Patterns on Reddit](https://mlsanity.com/paper/1703.05267/) (2017)

## Built on by (most cited first, 3 of 3)

- [Source Coverage and Citation Bias in LLM-based vs. Traditional Search Engines](https://mlsanity.com/paper/2512.09483/) (2025)
- [Predictively Combatting Toxicity in Health-related Online Discussions through Machine Learning](https://mlsanity.com/paper/2505.17068/) (2025)
- [Agent-based Modeling: Equilibrium, Echo Chambers, and Efficiency in Hybrid Coevolutionary Opinion Games](https://mlsanity.com/paper/2609.27639/) (2026)

## Abstract (arXiv)

Recent decisions to discontinue access to social media APIs are having detrimental effects on Internet research and the field of computational social science as a whole. This lack of access to data has been dubbed the Post-API era of Internet research. Fortunately, popular search engines have the means to crawl, capture, and surface social media data on their Search Engine Results Pages (SERP) if provided the proper search query, and may provide a solution to this dilemma. In the present work we ask: does SERP provide a complete and unbiased sample of social media data? Is SERP a viable alternative to direct API-access? To answer these questions, we perform a comparative analysis between (Google) SERP results and nonsampled data from Reddit and Twitter/X. We find that SERP results are highly biased in favor of popular posts; against political, pornographic, and vulgar posts; are more positive in their sentiment; and have large topical gaps. Overall, we conclude that SERP is not a viable alternative to social media API access.