mlsanityData to 7 Oct 2026

Implicit Two-Tower Policies

arXiv 2208.01191 ↗First posted 2 Aug 2022cs.LG cs.AI cs.NEcited by 3

Not on the map. Why?

Abstract (arXiv)

We present a new class of structured reinforcement learning policy-architectures, Implicit Two-Tower (ITT) policies, where the actions are chosen based on the attention scores of their learnable latent representations with those of the input states. By explicitly disentangling action from state processing in the policy stack, we achieve two main goals: substantial computational gains and better performance. Our architectures are compatible with both: discrete and continuous action spaces. By conducting tests on 15 environments from OpenAI Gym and DeepMind Control Suite, we show that ITT-architectures are particularly suited for blackbox/evolutionary optimization and the corresponding policy training algorithms outperform their vanilla unstructured implicit counterparts as well as commonly used explicit policies. We complement our analysis by showing how techniques such as hashing and lazy tower updates, critically relying on the two-tower structure of ITTs, can be applied to obtain additional computational improvements.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 28 of its 58 references that are in the map. First posted on arXiv 2 Aug 2022.

Builds on

28 of 58 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2016OpenAI Gymcited by 1,684
  6. 2020Rethinking Attention with Performerscited by 1,584
  7. 2018DeepMind Control Suitecited by 935
  8. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  10. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  11. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  12. 2020Hopfield Networks is All You Needcited by 461
16 more
  1. 2021Implicit Behavioral Cloningcited by 442
  2. 2021How to Train Your Energy-Based Modelscited by 271
  3. 2016A Theory of Generative ConvNetcited by 174
  4. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
  5. 2018Structured Evolution with Compact Architectures for Scalable Policy Optimizationcited by 48
  6. 2017The Unreasonable Effectiveness of Structured Random Orthogonal Embeddingscited by 43
  7. 2019Sampled Softmax with Random Fourier Featurescited by 33
  8. 2019Model Based Planning with Energy Based Modelscited by 32
  9. 2021Hybrid Random Featurescited by 25
  10. 2014On Symmetric and Asymmetric LSHs for Inner Product Searchcited by 23
  11. 2020From Pixels to Legs: Hierarchical Learning of Quadruped Locomotioncited by 20
  12. 2014Improved Asymmetric Locality Sensitive Hashing (ALSH) for Maximum Inner Product Search (MIPS)cited by 15
  13. 2019Energy-Based Continuous Inverse Optimal Controlcited by 14
  14. 2022Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradientscited by 12
  15. 2021Generalization Bounded Implicit Learning of Nearly Discontinuous Functionscited by 3
  16. 2020Sublinear Maximum Inner Product Search using Concomitants of Extreme Order Statisticscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 0; 2026: 1.

3 papers.

  1. 2024Towards Foundation-model-based Multiagent System to Accelerate AI for Social Impactcited by 9
  2. 2023Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learningcited by 0
  3. 2026The Evidence Ladder for Reinforcement Learning in Healthcare: From Retrospective Policies to Trusted Interventionscited by 0