# DuConTE: Dual-Granularity Text Encoder with Topology-Constrained Attention for Text-attributed Graphs

arXiv: https://arxiv.org/abs/2604.17411 · HTML: https://mlsanity.com/paper/2604.17411/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 34 references that are in the map.
- First posted on arXiv 19 Apr 2026.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [E(n) Equivariant Graph Neural Networks](https://mlsanity.com/paper/2102.09844/) (2021)
- [Recipe for a General, Powerful, Scalable Graph Transformer](https://mlsanity.com/paper/2205.12454/) (2022)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification](https://mlsanity.com/paper/2306.08385/) (2023)
- [LLaGA: Large Language and Graph Assistant](https://mlsanity.com/paper/2402.08170/) (2024)
- [Graph-Bert: Only Attention is Needed for Learning Graph Representations](https://mlsanity.com/paper/2001.05140/) (2020)
- [Learning on Large-scale Text-attributed Graphs via Variational Inference](https://mlsanity.com/paper/2210.14709/) (2022)
- [GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph](https://mlsanity.com/paper/2105.02605/) (2021)
- [Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction](https://mlsanity.com/paper/2111.00064/) (2021)
- [Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning](https://mlsanity.com/paper/2305.19523/) (2023)
- [GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner](https://mlsanity.com/paper/2304.04779/) (2023)
- [SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning](https://mlsanity.com/paper/2308.02565/) (2023)
- [Label-free Node Classification on Graphs with Large Language Models (LLMS)](https://mlsanity.com/paper/2310.04668/) (2023)
- [Efficient Tuning and Inference for Large Language Models on Textual Graphs](https://mlsanity.com/paper/2401.15569/) (2024)
- [Graph Transformers: A Survey](https://mlsanity.com/paper/2407.09777/) (2024)
- [Heterformer: Transformer-based Deep Node Representation Learning on Heterogeneous Text-Rich Networks](https://mlsanity.com/paper/2205.10282/) (2022)
- [Can LLMs Convert Graphs to Text-Attributed Graphs?](https://mlsanity.com/paper/2412.10136/) (2024)
- [Learning Long Range Dependencies on Graphs via Random Walks](https://mlsanity.com/paper/2406.03386/) (2024)
- [Bridging Local Details and Global Context in Text-Attributed Graphs](https://mlsanity.com/paper/2406.12608/) (2024)
- [Unifying Structured Data as Graph for Data-to-Text Pre-Training](https://mlsanity.com/paper/2401.01183/) (2024)
- [Pre-Training and Prompting for Few-Shot Node Classification on Text-Attributed Graphs](https://mlsanity.com/paper/2407.15431/) (2024)
- [Unleashing the Potential of Text-attributed Graphs: Automatic Relation Decomposition via Large Language Models](https://mlsanity.com/paper/2405.18581/) (2024)
- [UGMAE: A Unified Framework for Graph Masked Autoencoders](https://mlsanity.com/paper/2402.08023/) (2024)

## Abstract (arXiv)

Text-attributed graphs integrate semantic information of node texts with topological structure, offering significant value in various applications such as document classification and information extraction. Existing approaches typically encode textual content using language models (LMs), followed by graph neural networks (GNNs) to process structural information. However, during the LM-based text encoding phase, most methods not only perform semantic interaction solely at the word-token granularity, but also neglect the structural dependencies among texts from different nodes. In this work, we propose DuConTE, a dual-granularity text encoder with topology-constrained attention. The model employs a cascaded architecture of two pretrained LMs, encoding semantics first at the word-token granularity and then at the node granularity. During the self-attention computation in each LM, we dynamically adjust the attention mask matrix based on node connectivity, guiding the model to learn semantic correlations informed by the graph structure. Furthermore, when composing node representations from word-token embeddings, we separately evaluate the importance of tokens under the center-node context and the neighborhood context, enabling the capture of more contextually relevant semantic information. Extensive experiments on multiple benchmark datasets demonstrate that DuConTE achieves state-of-the-art performance on the majority of them.