# terms.txt: A Consent and Compensation Protocol for Agentic Web Access

arXiv: https://arxiv.org/abs/2609.11152 · HTML: https://mlsanity.com/paper/2609.11152/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 2 of its 19 references that are in the map.
- First posted on arXiv 10 Sep 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 2 of 2)

- [Consent in Crisis: The Rapid Decline of the AI Data Commons](https://mlsanity.com/paper/2407.14933/) (2024)
- [ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet](https://mlsanity.com/paper/2505.07834/) (2025)

## Built on by (most cited first, 1 of 1)

- [Get In, Decide, Prove: A Merchant-Side Readiness Benchmark for Agentic Commerce](https://mlsanity.com/paper/2610.17378/) (2026)

## Abstract (arXiv)

The open web ran on an unwritten bargain: sites admitted crawlers, and search engines sent visitors back. Public measurements show that bargain breaking under AI crawlers and agents. Automated clients now make up most requests, training dominates Cloudflare-classified crawling, and the largest AI platforms fetch thousands of pages for each visitor they return. The web's common control, robots.txt, cannot express identity, purpose, terms, or price, can be circumvented, and newer alternatives are largely proprietary CDN features. We specify terms.txt, a robots.txt-style file for per-path, per-purpose machine-access terms, plus an origin-enforced exchange using Web Bot Auth signatures, signed intent, delegation tokens, HTTP 402 negotiation, and signed receipts. We define what the exchange can enforce, audit, and leave to contract. A dependency-free implementation adds 0.20 to 0.65 ms per request on one vCPU.