# Affordance Representation and Recognition for Autonomous Agents

arXiv: https://arxiv.org/abs/2510.24459 · HTML: https://mlsanity.com/paper/2510.24459/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 28 Oct 2025.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)
- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 13 of 13)

- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis](https://mlsanity.com/paper/2307.12856/) (2023)
- [AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents](https://mlsanity.com/paper/2410.13825/) (2024)
- [A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models](https://mlsanity.com/paper/2503.23350/) (2025)
- [Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems](https://mlsanity.com/paper/2407.13032/) (2024)
- [HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systems](https://mlsanity.com/paper/2411.02959/) (2024)
- [Build the web for agents, not agents for the web](https://mlsanity.com/paper/2506.10953/) (2025)
- [WEPO: Web Element Preference Optimization for LLM-based Web Navigation](https://mlsanity.com/paper/2412.10742/) (2024)
- [Towards Cognitive Bots: Architectural Research Challenges](https://mlsanity.com/paper/2305.17308/) (2023)
- [Modeling Adaptive Self-healing Systems](https://mlsanity.com/paper/2304.12773/) (2023)
- [User-Like Bots for Cognitive Automation: A Survey](https://mlsanity.com/paper/2311.12154/) (2023)
- [NEXT-EVAL: Next Evaluation of Traditional and LLM Web Data Record Extraction](https://mlsanity.com/paper/2505.17125/) (2025)

## Built on by (most cited first, 2 of 2)

- [A Pattern Language for Resilient Visual Agents](https://mlsanity.com/paper/2604.28001/) (2026)
- [A Technical Taxonomy of LLM Agent Communication Protocols](https://mlsanity.com/paper/2606.19135/) (2026)

## Abstract (arXiv)

The autonomy of software agents is fundamentally dependent on their ability to construct an actionable internal world model from the structured data that defines their digital environment, such as the Document Object Model (DOM) of web pages and the semantic descriptions of web services. However, constructing this world model from raw structured data presents two critical challenges: the verbosity of raw HTML makes it computationally intractable for direct use by foundation models, while the static nature of hardcoded API integrations prevents agents from adapting to evolving services. This paper introduces a pattern language for world modeling from structured data, presenting two complementary architectural patterns. The DOM Transduction Pattern addresses the challenge of web page complexity by distilling} a verbose, raw DOM into a compact, task-relevant representation or world model optimized for an agent's reasoning core. Concurrently, the Hypermedia Affordances Recognition Pattern enables the agent to dynamically enrich its world model by parsing standardized semantic descriptions to discover and integrate the capabilities of unknown web services at runtime. Together, these patterns provide a robust framework for engineering agents that can efficiently construct and maintain an accurate world model, enabling scalable, adaptive, and interoperable automation across the web and its extended resources.