# Hierarchical Graph Topic Modeling with Topic Tree-based Transformer

arXiv: https://arxiv.org/abs/2502.11345 · HTML: https://mlsanity.com/paper/2502.11345/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 40 of its 83 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Hyperbolic Graph Convolutional Neural Networks](https://mlsanity.com/paper/1910.12933/) (2019)
- [Graph Convolutional Networks for Text Classification](https://mlsanity.com/paper/1809.05679/) (2018)
- [Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry](https://mlsanity.com/paper/1806.03417/) (2018)
- [Hyperbolic Graph Neural Networks](https://mlsanity.com/paper/1910.12892/) (2019)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [LinkBERT: Pretraining Language Models with Document Links](https://mlsanity.com/paper/2203.15827/) (2022)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph](https://mlsanity.com/paper/2105.02605/) (2021)
- [Neural Variational Inference for Text Processing](https://mlsanity.com/paper/1511.06038/) (2015)
- [Constant Curvature Graph Convolutional Networks](https://mlsanity.com/paper/1911.05076/) (2019)
- [Be More with Less: Hypergraph Attention Networks for Inductive Text Classification](https://mlsanity.com/paper/2011.00387/) (2020)
- [Hyperbolic Graph Attention Network](https://mlsanity.com/paper/1912.03046/) (2019)
- [Discrete-time Temporal Network Embedding via Implicit Hierarchical Learning in Hyperbolic Space](https://mlsanity.com/paper/2107.03767/) (2021)
- [Augmenting Low-Resource Text Classification with Graph-Grounded Pre-training and Prompting](https://mlsanity.com/paper/2305.03324/) (2023)
- [Patton: Language Model Pretraining on Text-Rich Networks](https://mlsanity.com/paper/2305.12268/) (2023)
- [Effective Neural Topic Modeling with Embedding Clustering Regularization](https://mlsanity.com/paper/2306.04217/) (2023)
- [Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning](https://mlsanity.com/paper/2211.12878/) (2022)
- [FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model](https://mlsanity.com/paper/2405.17978/) (2024)
- [Heterformer: Transformer-based Deep Node Representation Learning on Heterogeneous Text-Rich Networks](https://mlsanity.com/paper/2205.10282/) (2022)
- [InfoCTM: A Mutual Information Maximization Perspective of Cross-Lingual Topic Modeling](https://mlsanity.com/paper/2304.03544/) (2023)

## Abstract (arXiv)

Textual documents are commonly connected in a hierarchical graph structure where a central document links to others with an exponentially growing connectivity. Though Hyperbolic Graph Neural Networks (HGNNs) excel at capturing such graph hierarchy, they cannot model the rich textual semantics within documents. Moreover, text contents in documents usually discuss topics of different specificity. Hierarchical Topic Models (HTMs) discover such latent topic hierarchy within text corpora. However, most of them focus on the textual content within documents, and ignore the graph adjacency across interlinked documents. We thus propose a Hierarchical Graph Topic Modeling Transformer to integrate both topic hierarchy within documents and graph hierarchy across documents into a unified Transformer. Specifically, to incorporate topic hierarchy within documents, we design a topic tree and infer a hierarchical tree embedding for hierarchical topic modeling. To preserve both topic and graph hierarchies, we design our model in hyperbolic space and propose Hyperbolic Doubly Recurrent Neural Network, which models ancestral and fraternal tree structure. Both hierarchies are inserted into each Transformer layer to learn unified representations. Both supervised and unsupervised experiments verify the effectiveness of our model.