# Conditional Set Generation with Transformers

arXiv: https://arxiv.org/abs/2006.16841 · HTML: https://mlsanity.com/paper/2006.16841/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jun 2020.

## Built on by (most cited first, 30 of 32)

- [Equivariant Diffusion for Molecule Generation in 3D](https://mlsanity.com/paper/2203.17003/) (2022)
- [ATISS: Autoregressive Transformers for Indoor Scene Synthesis](https://mlsanity.com/paper/2110.03675/) (2021)
- [Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation](https://mlsanity.com/paper/2104.01148/) (2021)
- [SetVAE: Learning Hierarchical Composition for Generative Modeling of Set-Structured Data](https://mlsanity.com/paper/2103.15619/) (2021)
- [Complex-Valued Autoencoders for Object Discovery](https://mlsanity.com/paper/2204.02075/) (2022)
- [Top-N: Equivariant set and graph generation without exchangeability](https://mlsanity.com/paper/2110.02096/) (2021)
- [CaloClouds: Fast Geometry-Independent Highly-Granular Calorimeter Simulation](https://mlsanity.com/paper/2305.04847/) (2023)
- [Rotating Features for Object Discovery](https://mlsanity.com/paper/2306.00600/) (2023)
- [A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2106.02097/) (2021)
- [EPiC-GAN: Equivariant Point Cloud Generation for Particle Jets](https://mlsanity.com/paper/2301.08128/) (2023)
- [Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation](https://mlsanity.com/paper/2111.12193/) (2021)
- [Prefix-Tree Decoding for Predicting Mass Spectra from Molecules](https://mlsanity.com/paper/2303.06470/) (2023)
- [multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning](https://mlsanity.com/paper/2106.01354/) (2021)
- [Object-Centric Learning with Slot Mixture Module](https://mlsanity.com/paper/2311.04640/) (2023)
- [Learning Neural Set Functions Under the Optimal Subset Oracle](https://mlsanity.com/paper/2203.01693/) (2022)
- [Conditional set generation using Seq2seq models](https://mlsanity.com/paper/2205.12485/) (2022)
- [Robust and Controllable Object-Centric Learning through Energy-based Models](https://mlsanity.com/paper/2210.05519/) (2022)
- [Let There Be Order: Rethinking Ordering in Autoregressive Graph Generation](https://mlsanity.com/paper/2305.15562/) (2023)
- [Set-to-Sequence Methods in Machine Learning: a Review](https://mlsanity.com/paper/2103.09656/) (2021)
- [Audio synthesizer inversion in symmetric parameter spaces with approximately equivariant flow matching](https://mlsanity.com/paper/2506.07199/) (2025)
- [AudioSlots: A slot-centric generative model for audio separation](https://mlsanity.com/paper/2305.05591/) (2023)
- [Binding Dynamics in Rotating Features](https://mlsanity.com/paper/2402.05627/) (2024)
- [Set Representation Learning with Generalized Sliced-Wasserstein Embeddings](https://mlsanity.com/paper/2103.03892/) (2021)
- [Permutation-Invariant Set Autoencoders with Fixed-Size Embeddings for Multi-Agent Learning](https://mlsanity.com/paper/2302.12826/) (2023)
- [Advances in Set Function Learning: A Survey of Techniques and Applications](https://mlsanity.com/paper/2501.14991/) (2025)
- [Search for Concepts: Discovering Visual Concepts Using Direct Optimization](https://mlsanity.com/paper/2210.14808/) (2022)
- [Sequence-to-Set Generative Models](https://mlsanity.com/paper/2209.08801/) (2022)
- [Image To Tree with Recursive Prompting](https://mlsanity.com/paper/2301.00447/) (2023)
- [Outfit Completion via Conditional Set Transformation](https://mlsanity.com/paper/2311.16630/) (2023)
- [Tokenize Image as a Set](https://mlsanity.com/paper/2503.16425/) (2025)

## Abstract (arXiv)

A set is an unordered collection of unique elements--and yet many machine learning models that generate sets impose an implicit or explicit ordering. Since model performance can depend on the choice of order, any particular ordering can lead to sub-optimal results. An alternative solution is to use a permutation-equivariant set generator, which does not specify an order-ing. An example of such a generator is the DeepSet Prediction Network (DSPN). We introduce the Transformer Set Prediction Network (TSPN), a flexible permutation-equivariant model for set prediction based on the transformer, that builds upon and outperforms DSPN in the quality of predicted set elements and in the accuracy of their predicted sizes. We test our model on MNIST-as-point-clouds (SET-MNIST) for point-cloud generation and on CLEVR for object detection.