mlsanityData to 14 Oct 2026

Learning Graph Quantized Tokenizers

arXiv 2410.13798 ↗First posted 17 Oct 2024cs.NE cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Transformers serve as the backbone architectures of Foundational Models, where domain-specific tokenizers allow them to adapt to various domains. Graph Transformers (GTs) have recently emerged as leading models in geometric deep learning, outperforming Graph Neural Networks (GNNs) in various graph learning tasks. However, the development of tokenizers for graphs has lagged behind other modalities. To address this, we introduce GQT (Graph Quantized Tokenizer), which decouples tokenizer training from Transformer training by leveraging multi-task graph self-supervised learning, yielding robust and generalizable graph tokens. Furthermore, the GQT utilizes Residual Vector Quantization (RVQ) to learn hierarchical discrete tokens, resulting in significantly reduced memory requirements and improved generalization capabilities. By combining the GQT with token modulation, a Transformer encoder achieves state-of-the-art performance on 20 out of 22 benchmarks, including large-scale homophilic and heterophilic datasets.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 96 of its 115 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

96 of 115 references.

References · 96

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2024The Llama 3 Herd of Modelscited by 14,715
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  9. 2017Graph Attention Networkscited by 8,923
  10. 2017Inductive Representation Learning on Large Graphscited by 6,320
  11. 2017Neural Discrete Representation Learningcited by 4,712
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
38 more
  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  5. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  6. 2020Graph Contrastive Learning with Augmentationscited by 1,208
  7. 2018Deep Graph Infomaxcited by 1,106
  8. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  9. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  10. 2018Pitfalls of Graph Neural Network Evaluationcited by 1,007
  11. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 993
  12. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 976
  13. 2019Strategies for Pre-training Graph Neural Networkscited by 971
  14. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  15. 2020Geom-GCN: Geometric Graph Convolutional Networkscited by 826
  16. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  17. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 713
  18. 2015Image-based Recommendations on Styles and Substitutescited by 706
  19. 2020A Generalization of Transformer Networks to Graphscited by 661
  20. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  21. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  22. 2018Junction Tree Variational Autoencoder for Molecular Graph Generationcited by 652
  23. 2020Contrastive Multi-View Representation Learning on Graphscited by 652
  24. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  25. 2020Adaptive Universal Generalized PageRank Graph Neural Networkcited by 595
  26. 2019Multi-scale Attributed Node Embeddingcited by 548
  27. 2020Jukebox: A Generative Model for Musiccited by 542
  28. 2020Deep Graph Contrastive Representation Learningcited by 539
  29. 2021Rethinking Graph Transformers with Spectral Attentioncited by 473
  30. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  31. 2022GraphMAE: Self-Supervised Masked Graph Autoencoderscited by 448
  32. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  33. 2023A critical look at the evaluation of GNNs under heterophily: Are we really making progress?cited by 343
  34. 2017Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Rankingcited by 335
  35. 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 333
  36. 2021Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methodscited by 311
  37. 2021Graph Contrastive Learning Automatedcited by 287
  38. 2020SIGN: Scalable Inception Graph Neural Networkscited by 284

50 most cited of 96 All 96 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 7.

19 papers.

Sort:

By area:

  1. 2025FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenizationcited by 22
  2. 2025Graph4MM: Weaving Multimodal Learning with Structural Informationcited by 17
  3. 2025Towards Effective Federated Graph Foundation Model via Mitigating Knowledge Entanglementcited by 14
  4. 2025Breaking Silos: Adaptive Model Fusion Unlocks Better Time Series Forecastingcited by 12
  5. 2026MM-OpenFGL: A Comprehensive Benchmark for Multimodal Federated Graph Learningcited by 11
  6. 2024Graph Learning in the Era of LLMs: A Survey from the Perspective of Data, Models, and Taskscited by 10
  7. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 8
  8. 2025Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapsecited by 7
  9. 2026Structure-Centric Graph Foundation Model via Geometric Basescited by 5
  10. 2025Learnable Spatial-Temporal Positional Encoding for Link Predictioncited by 3
  11. 2025HeroFilter: Adaptive Spectral Graph Filter for Varying Heterophilic Relationscited by 3
  12. 2025Graph is a Natural Regularization: Revisiting Vector Quantization for Graph Representation Learningcited by 2
7 more
  1. 2025GraphTARIF: Linear Graph Transformer with Augmented Rank and Improved Focuscited by 2
  2. 2026OWLEYE: Zero-Shot Learner for Cross-Domain Graph Data Anomaly Detectioncited by 2
  3. 2026Rethinking Federated Graph Foundation Models: A Graph-Language Alignment-based Approachcited by 2
  4. 2026VesselTok: Tokenizing Vessel-like 3D Biomedical Graph Representations for Reconstruction and Generationcited by 2
  5. 2026SCGFM-ART: Amortized Relational Transport for Structure-Centric Graph Foundation Modelscited by 1
  6. 2025Two Facets of the Same Optimization Coin: Model Degradation and Representation Collapse in Graph Foundation Modelscited by 0
  7. 2026KoRe: Compact Knowledge Representations for Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Low-bit Quantization for Deep Graph Neural Networks with Smoothness-aware Message Propagation
  2. 2025Hierarchical Vector Quantized Graph Autoencoder with Annealing-Based Code Selection
  3. 2026VecFormer: Towards Efficient and Generalizable Graph Transformer with Graph Token Attention
  4. 2025A Survey of Quantized Graph Representation Learning: Connecting Graph Structures with Large Language Models
  5. 2024SGFormer: Single-Layer Graph Transformers with Approximation-Free Linear Complexity
  6. 2025Rethinking Tokenized Graph Transformers for Node Classification
  7. 2024SpinQuant: LLM quantization with learned rotations
  8. 2022GHN-Q: Parameter Prediction for Unseen Quantized Convolutional Architectures via Graph Hypernetworks