# DevFormer: A Symmetric Transformer for Context-Aware Device Placement

arXiv: https://arxiv.org/abs/2205.13225 · HTML: https://mlsanity.com/paper/2205.13225/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 40 references that are in the map.
- First posted on arXiv 26 May 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [Attention, Learn to Solve Routing Problems!](https://mlsanity.com/paper/1803.08475/) (2018)
- [Neural Combinatorial Optimization with Reinforcement Learning](https://mlsanity.com/paper/1611.09940/) (2016)
- [Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation](https://mlsanity.com/paper/2106.04399/) (2021)
- [POMO: Policy Optimization with Multiple Optima for Reinforcement Learning](https://mlsanity.com/paper/2010.16011/) (2020)
- [Contextual Markov Decision Processes](https://mlsanity.com/paper/1502.02259/) (2015)
- [DRiLLS: Deep Reinforcement Learning for Logic Synthesis](https://mlsanity.com/paper/1911.04021/) (2019)
- [On Joint Learning for Solving Placement and Routing in Chip Design](https://mlsanity.com/paper/2111.00234/) (2021)
- [A Deep Reinforcement Learning Approach for Global Routing](https://mlsanity.com/paper/1906.08809/) (2019)
- [Transformer Network-based Reinforcement Learning Method for Power Distribution Network (PDN) Optimization of High Bandwidth Memory (HBM)](https://mlsanity.com/paper/2203.15722/) (2022)
- [Track-Assignment Detailed Routing Using Attention-based Policy Model With Supervision](https://mlsanity.com/paper/2010.13702/) (2020)

## Built on by (most cited first, 11 of 11)

- [ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution](https://mlsanity.com/paper/2402.01145/) (2024)
- [RL4CO: an Extensive Reinforcement Learning for Combinatorial Optimization Benchmark](https://mlsanity.com/paper/2306.17100/) (2023)
- [Ant Colony Sampling with GFlowNets for Combinatorial Optimization](https://mlsanity.com/paper/2403.07041/) (2024)
- [Symmetric Replay Training: Enhancing Sample Efficiency in Deep Reinforcement Learning for Combinatorial Optimization](https://mlsanity.com/paper/2306.01276/) (2023)
- [TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agents](https://mlsanity.com/paper/2404.12530/) (2024)
- [PARCO: Parallel AutoRegressive Models for Multi-Agent Combinatorial Optimization](https://mlsanity.com/paper/2409.03811/) (2024)
- [RoutePlacer: An End-to-End Routability-Aware Placer with Graph Neural Network](https://mlsanity.com/paper/2406.02651/) (2024)
- [Attention-Based Deep Reinforcement Learning for Qubit Allocation in Modular Quantum Architectures](https://mlsanity.com/paper/2406.11452/) (2024)
- [TransPlace: Transferable Circuit Global Placement via Graph Neural Network](https://mlsanity.com/paper/2501.05667/) (2025)
- [ReflectivePrompt: Reflective evolution in autoprompting algorithms](https://mlsanity.com/paper/2508.18870/) (2025)
- [PhysEDA: Physics-Aware Learning Framework for Efficient EDA With Manhattan Distance Decay](https://mlsanity.com/paper/2605.10547/) (2026)

## Abstract (arXiv)

In this paper, we present DevFormer, a novel transformer-based architecture for addressing the complex and computationally demanding problem of hardware design optimization. Despite the demonstrated efficacy of transformers in domains including natural language processing and computer vision, their use in hardware design has been limited by the scarcity of offline data. Our approach addresses this limitation by introducing strong inductive biases such as relative positional embeddings and action-permutation symmetricity that effectively capture the hardware context and enable efficient design optimization with limited offline data. We apply DevFoemer to the problem of decoupling capacitor placement and show that it outperforms state-of-the-art methods in both simulated and real hardware, leading to improved performances while reducing the number of components by more than 30%. Finally, we show that our approach achieves promising results in other offline contextual learning-based combinatorial optimization tasks.