# Improving global awareness of linkset predictions using Cross-Attentive Modulation tokens

arXiv: https://arxiv.org/abs/2405.19375 · HTML: https://mlsanity.com/paper/2405.19375/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Benchmarking Graph Neural Networks](https://mlsanity.com/paper/2003.00982/) (2020)
- [A Generalization of Transformer Networks to Graphs](https://mlsanity.com/paper/2012.09699/) (2020)
- [MolGAN: An implicit generative model for small molecular graphs](https://mlsanity.com/paper/1805.11973/) (2018)
- [GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders](https://mlsanity.com/paper/1802.03480/) (2018)
- [GraphAF: a Flow-based Autoregressive Model for Molecular Graph Generation](https://mlsanity.com/paper/2001.09382/) (2020)
- [GraphGAN: Graph Representation Learning with Generative Adversarial Nets](https://mlsanity.com/paper/1711.08267/) (2017)
- [Global Context Enhanced Graph Neural Networks for Session-based Recommendation](https://mlsanity.com/paper/2106.05081/) (2021)
- [GNN-FiLM: Graph Neural Networks with Feature-wise Linear Modulation](https://mlsanity.com/paper/1906.12192/) (2019)
- [GCN-GAN: A Non-linear Temporal Link Prediction Model for Weighted Dynamic Networks](https://mlsanity.com/paper/1901.09165/) (2019)

## Abstract (arXiv)

This work introduces Cross-Attentive Modulation (CAM) tokens, which are tokens whose initial value is learned, gather information through cross-attention, and modulate the nodes and edges accordingly. These tokens are meant to improve the global awareness of link predictions models which, based on graph neural networks, can struggle to capture graph-level features. This lack of ability to feature high level representations is particularly limiting when predicting multiple or entire sets of links. We implement CAM tokens in a simple attention-based link prediction model and in a graph transformer, which we also use in a denoising diffusion framework. A brief introduction to our toy datasets will then be followed by benchmarks which prove that CAM token improve the performance of the model they supplement and outperform a baseline with diverse statistical graph attributes.