# Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers

arXiv: https://arxiv.org/abs/2411.15091 · HTML: https://mlsanity.com/paper/2411.15091/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2024.

## Built on by (most cited first, 7 of 7)

- [Web Intellectual Property at Risk: Preventing Unauthorized Real-Time Retrieval by Large Language Models](https://mlsanity.com/paper/2505.12655/) (2025)
- [Developer Experience with AI Coding Agents: HTTP Behavioral Signatures in Documentation Portals](https://mlsanity.com/paper/2604.02544/) (2026)
- [Whose Agent Are You? Multi-Layer Fingerprinting and Attribution of Autonomous Web Agents](https://mlsanity.com/paper/2606.20910/) (2026)
- [Identifying AI Web Scrapers Using Canary Tokens](https://mlsanity.com/paper/2605.13706/) (2026)
- [The Agentic Web Requires New Normative Infrastructure](https://mlsanity.com/paper/2606.10711/) (2026)
- [On the Internet, Nobody Knows You're an LLM Bot: Unmasking Web Agents with Multi-Layer Fingerprinting](https://mlsanity.com/paper/2606.30119/) (2026)
- [Get In, Decide, Prove: A Merchant-Side Readiness Benchmark for Agentic Commerce](https://mlsanity.com/paper/2610.17378/) (2026)

## Abstract (arXiv)

The success of generative AI relies heavily on training on data scraped through extensive crawling of the Internet, a practice that has raised significant copyright, privacy, and ethical concerns. While few measures are designed to resist a resource-rich adversary determined to scrape a site, crawlers can be impacted by a range of existing tools such as robots.txt, NoAI meta tags, and active crawler blocking by reverse proxies. In this work, we seek to understand the ability and efficacy of today's networking tools to protect content creators against AI-related crawling. For targeted populations like human artists, do they have the technical knowledge and agency to utilize crawler-blocking tools such as robots.txt, and can such tools be effective? Using large scale measurements and a targeted user study of 203 professional artists, we find strong demand for tools like robots.txt, but significantly constrained by critical hurdles in technical awareness, agency in deploying them, and limited efficacy against unresponsive crawlers. We further test and evaluate network-level crawler blockers provided by reverse proxies. Despite relatively limited deployment today, they offer stronger protections against AI crawlers, but still come with their own set of limitations.