# Modelling non-reinforced preferences using selective attention

arXiv: https://arxiv.org/abs/2207.13699 · HTML: https://mlsanity.com/paper/2207.13699/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 25 Jul 2022.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Planning to Explore via Self-Supervised World Models](https://mlsanity.com/paper/2005.05960/) (2020)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments](https://mlsanity.com/paper/2002.12292/) (2020)
- [Model-Based Active Exploration](https://mlsanity.com/paper/1810.12162/) (2018)
- [Active inference on discrete state-spaces: a synthesis](https://mlsanity.com/paper/2001.07203/) (2020)
- [Deep active inference agents using Monte-Carlo methods](https://mlsanity.com/paper/2006.04176/) (2020)
- [Ready Policy One: World Building Through Active Learning](https://mlsanity.com/paper/2002.02693/) (2020)
- [Exploration and preference satisfaction trade-off in reward-free learning](https://mlsanity.com/paper/2106.04316/) (2021)
- [Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration](https://mlsanity.com/paper/1903.07400/) (2019)
- [Active inference, Bayesian optimal design, and expected utility](https://mlsanity.com/paper/2110.04074/) (2021)
- [Geometric Methods for Sampling, Optimisation, Inference and Adaptive Agents](https://mlsanity.com/paper/2203.10592/) (2022)
- [Reclaiming saliency: rhythmic precision-modulated action and perception](https://mlsanity.com/paper/2203.12652/) (2022)

## Built on by (most cited first, 1 of 1)

- [On efficient computation in active inference](https://mlsanity.com/paper/2307.00504/) (2023)

## Abstract (arXiv)

How can artificial agents learn non-reinforced preferences to continuously adapt their behaviour to a changing environment? We decompose this question into two challenges: (i) encoding diverse memories and (ii) selectively attending to these for preference formation. Our proposed non-reinforced preference learning mechanism using selective attention, textscNore, addresses both by leveraging the agent's world model to collect a diverse set of experiences which are interleaved with imagined roll-outs to encode memories. These memories are selectively attended to, using attention and gating blocks, to update agent's preferences. We validate textscNore in a modified OpenAI Gym FrozenLake environment (without any external signal) with and without volatility under a fixed model of the environment -- and compare its behaviour to textscPepper, a Hebbian preference learning mechanism. We demonstrate that textscNore provides a straightforward framework to induce exploratory preferences in the absence of external signals.