# A Semi-amortized Lifted Learning-to-Optimize Masked (SALLO-M) Transformer Model for Scalable and Generalizable Beamforming

arXiv: https://arxiv.org/abs/2510.13077 · HTML: https://mlsanity.com/paper/2510.13077/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 15 Oct 2025.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [What Algorithms can Transformers Learn? A Study in Length Generalization](https://mlsanity.com/paper/2310.16028/) (2023)
- [Learned Optimizers that Scale and Generalize](https://mlsanity.com/paper/1703.04813/) (2017)
- [Semi-Amortized Variational Autoencoders](https://mlsanity.com/paper/1802.02550/) (2018)
- [Iterative Algorithm Induced Deep-Unfolding Neural Networks: Precoding Design for Multiuser MIMO Systems](https://mlsanity.com/paper/2006.08099/) (2020)
- [Learning to Optimize Neural Nets](https://mlsanity.com/paper/1703.00441/) (2017)
- [Mask Attention Networks: Rethinking and Strengthen Transformer](https://mlsanity.com/paper/2103.13597/) (2021)
- [Deep Learning for Distributed Channel Feedback and Multiuser Precoding in FDD Massive MIMO](https://mlsanity.com/paper/2007.06512/) (2020)
- [Learning-based Predictive Beamforming for Integrated Sensing and Communication in Vehicular Networks](https://mlsanity.com/paper/2108.11540/) (2021)
- [Learn to Rapidly and Robustly Optimize Hybrid Precoding](https://mlsanity.com/paper/2301.00369/) (2023)
- [Hybrid Precoding for Multi-User Millimeter Wave Massive MIMO Systems: A Deep Learning Approach](https://mlsanity.com/paper/1911.04239/) (2019)
- [A Bipartite Graph Neural Network Approach for Scalable Beamforming Optimization](https://mlsanity.com/paper/2207.05364/) (2022)
- [GNN-Based Beamforming for Sum-Rate Maximization in MU-MISO Networks](https://mlsanity.com/paper/2311.03659/) (2023)
- [HPE Transformer: Learning to Optimize Multi-Group Multicast Beamforming Under Nonconvex QoS Constraints](https://mlsanity.com/paper/2402.16081/) (2024)
- [Learning Precoding in Multi-user Multi-antenna Systems: Transformer or Graph Transformer?](https://mlsanity.com/paper/2503.02998/) (2025)
- [An Encoder-Decoder Network for Beamforming over Sparse Large-Scale MIMO Channels](https://mlsanity.com/paper/2510.02355/) (2025)
- [Minimizing Energy Consumption in MU-MIMO via Antenna Muting by Neural Networks with Asymmetric Loss](https://mlsanity.com/paper/2306.05162/) (2023)
- [Adaptive TTD Configurations for Near-Field Communications: An Unsupervised Transformer Approach](https://mlsanity.com/paper/2403.18146/) (2024)
- [BERT4beam: Large AI Model Enabled Generalized Beamforming Optimization](https://mlsanity.com/paper/2509.11056/) (2025)

## Built on by (most cited first, 2 of 2)

- [Self-Evolving In-Context Learning for Direct Pilot-to-Beamformer Design in MU-MISO Systems](https://mlsanity.com/paper/2607.11970/) (2026)
- [Learning-to-Transition for Large-scale and High-Order MIMO Detection](https://mlsanity.com/paper/2608.14511/) (2026)

## Abstract (arXiv)

We develop an unsupervised deep learning framework for real-time scalable and generalizable downlink beamforming in multi-user multiple-input single-output (MU-MISO) systems. The proposed semi-amortized lifted learning-to-optimize (SALLO) framework employs a multi-layer Transformer to iteratively refine an auxiliary variable and the beamformer solution, with a few projected gradient ascent steps at each layer. A key feature of our SALLO Transformer model is that it can handle varying numbers of users and antennas, enabled by a user-antenna dual tokenization and a structured sample/attention masking scheme, leading to generalization across different configurations without retraining. To improve convergence and robustness, we introduce three training strategies: (a) sliding-window training to stabilize gradient propagation, (b) curriculum learning with random masking to enable user-antenna configuration generalization and prevent poor early-stage convergence, and (c) sample replay to mitigate catastrophic forgetting during multi-stage training. Ablation studies validate several key architecture designs and show that the enhanced training scheme improves both generalizability and solution quality. Simulation results over both Gaussian and sparse channels show that the proposed scheme consistently outperforms existing deep learning baselines across diverse system configurations and channel conditions. The performance gain becomes more pronounced in overloaded regimes, highlighting improved robustness under challenging scenarios. Furthermore, our scheme surpasses the WMMSE benchmark in underloaded systems and even in overloaded systems when the overloading factor is below certain threshold. These gains are achieved with fast inference and a substantially more lightweight model than wireless foundation models.