# A Hyper-Transformer model for Controllable Pareto Front Learning with Split Feasibility Constraints

arXiv: https://arxiv.org/abs/2402.05955 · HTML: https://mlsanity.com/paper/2402.05955/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 75 references that are in the map.
- First posted on arXiv 4 Feb 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Are Transformers universal approximators of sequence-to-sequence functions?](https://mlsanity.com/paper/1912.10077/) (2019)
- [On the Relationship between Self-Attention and Convolutional Layers](https://mlsanity.com/paper/1911.03584/) (2019)
- [Hash Layers For Large Sparse Models](https://mlsanity.com/paper/2106.04426/) (2021)
- [Pareto Multi-Task Learning](https://mlsanity.com/paper/1912.12854/) (2019)
- [Learning the Pareto Front with Hypernetworks](https://mlsanity.com/paper/2010.04104/) (2020)
- [Approximating Continuous Functions by ReLU Nets of Minimal Width](https://mlsanity.com/paper/1710.11278/) (2017)
- [O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformers](https://mlsanity.com/paper/2006.04862/) (2020)
- [Controllable Pareto Multi-Task Learning](https://mlsanity.com/paper/2010.06313/) (2020)
- [Controllable Dynamic Multi-Task Architectures](https://mlsanity.com/paper/2203.14949/) (2022)
- [Can Vision Transformers Perform Convolution?](https://mlsanity.com/paper/2111.01353/) (2021)
- [Improving Pareto Front Learning via Multi-Sample Hypernetworks](https://mlsanity.com/paper/2212.01130/) (2022)
- [Controllable Multi-Objective Re-ranking with Policy Hypernetworks](https://mlsanity.com/paper/2306.05118/) (2023)
- [A Framework for Controllable Pareto Front Learning with Completed Scalarization Functions and its Applications](https://mlsanity.com/paper/2302.12487/) (2023)
- [A Monotonic Optimization Approach for Solving Strictly Quasiconvex Multiobjective Programming Problems](https://mlsanity.com/paper/2003.07549/) (2020)
- [Data-Driven Evolutionary Multi-Objective Optimization Based on Multiple-Gradient Descent for Disconnected Pareto Fronts](https://mlsanity.com/paper/2205.14344/) (2022)
- [Dynamic string-averaging CQ-methods for the split feasibility problem with percentage violation constraints arising in radiation therapy treatment planning](https://mlsanity.com/paper/1911.12041/) (2019)
- [Self-adaptive algorithms for quasiconvex programming and applications to machine learning](https://mlsanity.com/paper/2212.06379/) (2022)

## Built on by (most cited first, 3 of 3)

- [Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyond](https://mlsanity.com/paper/2501.10945/) (2025)
- [CAMEx: Curvature-aware Merging of Experts](https://mlsanity.com/paper/2502.18821/) (2025)
- [Efficient Hessian-Free Methods for Multi-Objective Bilevel Optimization with Nonconvex Lower Level](https://mlsanity.com/paper/2608.12704/) (2026)

## Abstract (arXiv)

Controllable Pareto front learning (CPFL) approximates the Pareto solution set and then locates a Pareto optimal solution with respect to a given reference vector. However, decision-maker objectives were limited to a constraint region in practice, so instead of training on the entire decision space, we only trained on the constraint region. Controllable Pareto front learning with Split Feasibility Constraints (SFC) is a way to find the best Pareto solutions to a split multi-objective optimization problem that meets certain constraints. In the previous study, CPFL used a Hypernetwork model comprising multi-layer perceptron (Hyper-MLP) blocks. With the substantial advancement of transformer architecture in deep learning, transformers can outperform other architectures in various tasks. Therefore, we have developed a hyper-transformer (Hyper-Trans) model for CPFL with SFC. We use the theory of universal approximation for the sequence-to-sequence function to show that the Hyper-Trans model makes MED errors smaller in computational experiments than the Hyper-MLP model.