# Learning to Guide and to Be Guided in the Architect-Builder Problem

arXiv: https://arxiv.org/abs/2112.07342 · HTML: https://mlsanity.com/paper/2112.07342/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 14 Dec 2021.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Self-Imitation Learning](https://mlsanity.com/paper/1806.05635/) (2018)
- [Learning to Understand Goal Specifications by Modelling Reward](https://mlsanity.com/paper/1806.01946/) (2018)
- [Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration](https://mlsanity.com/paper/2002.09253/) (2020)
- [Emergent Communication through Negotiation](https://mlsanity.com/paper/1804.03980/) (2018)
- [Emergent Social Learning via Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2010.00581/) (2020)
- [Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning](https://mlsanity.com/paper/1901.08492/) (2019)
- [Interactive Learning from Activity Description](https://mlsanity.com/paper/2102.07024/) (2021)
- [Learning to Interactively Learn and Assist](https://mlsanity.com/paper/1906.10187/) (2019)
- [NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment](https://mlsanity.com/paper/2110.06536/) (2021)
- [Interaction-Grounded Learning](https://mlsanity.com/paper/2106.04887/) (2021)

## Built on by (most cited first, 3 of 3)

- [A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem](https://mlsanity.com/paper/2305.17198/) (2023)
- [Contrastive Multimodal Learning for Emergence of Graphical Sensory-Motor Communication](https://mlsanity.com/paper/2210.06468/) (2022)
- [Self-Calibrating BCIs: Ranking and Recovery of Mental Targets Without Labels](https://mlsanity.com/paper/2506.11151/) (2025)

## Abstract (arXiv)

We are interested in interactive agents that learn to coordinate, namely, a builder -- which performs actions but ignores the goal of the task, i.e. has no access to rewards -- and an architect which guides the builder towards the goal of the task. We define and explore a formal setting where artificial agents are equipped with mechanisms that allow them to simultaneously learn a task while at the same time evolving a shared communication protocol. Ideally, such learning should only rely on high-level communication priors and be able to handle a large variety of tasks and meanings while deriving communication protocols that can be reused across tasks. We present the Architect-Builder Problem (ABP): an asymmetrical setting in which an architect must learn to guide a builder towards constructing a specific structure. The architect knows the target structure but cannot act in the environment and can only send arbitrary messages to the builder. The builder on the other hand can act in the environment, but receives no rewards nor has any knowledge about the task, and must learn to solve it relying only on the messages sent by the architect. Crucially, the meaning of messages is initially not defined nor shared between the agents but must be negotiated throughout learning. Under these constraints, we propose Architect-Builder Iterated Guiding (ABIG), a solution to ABP where the architect leverages a learned model of the builder to guide it while the builder uses self-imitation learning to reinforce its guided behavior. We analyze the key learning mechanisms of ABIG and test it in 2D tasks involving grasping cubes, placing them at a given location, or building various shapes. ABIG results in a low-level, high-frequency, guiding communication protocol that not only enables an architect-builder pair to solve the task at hand, but that can also generalize to unseen tasks.