# HiTeC: Hierarchical Contrastive Learning on Text-Attributed Hypergraph with Semantic-Aware Augmentation

arXiv: https://arxiv.org/abs/2508.03104 · HTML: https://mlsanity.com/paper/2508.03104/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Hypergraph Neural Networks](https://mlsanity.com/paper/1809.09401/) (2018)
- [Hypergraph Convolution and Hypergraph Attention](https://mlsanity.com/paper/1901.08150/) (2019)
- [Large-Scale Representation Learning on Graphs via Bootstrapping](https://mlsanity.com/paper/2102.06514/) (2021)
- [Learning on Large-scale Text-attributed Graphs via Variational Inference](https://mlsanity.com/paper/2210.14709/) (2022)
- [GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph](https://mlsanity.com/paper/2105.02605/) (2021)
- [Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction](https://mlsanity.com/paper/2111.00064/) (2021)
- [You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks](https://mlsanity.com/paper/2106.13264/) (2021)
- [Self-Supervised Hypergraph Transformer for Recommender Systems](https://mlsanity.com/paper/2207.14338/) (2022)
- [Augmentations in Hypergraph Contrastive Learning: Fabricated and Generative](https://mlsanity.com/paper/2210.03801/) (2022)
- [I'm Me, We're Us, and I'm Us: Tri-directional Contrastive Learning on Hypergraphs](https://mlsanity.com/paper/2206.04739/) (2022)
- [Hypernetwork Science via High-Order Hypergraph Walks](https://mlsanity.com/paper/1906.11295/) (2019)
- [HypeBoy: Generative Self-Supervised Representation Learning on Hypergraphs](https://mlsanity.com/paper/2404.00638/) (2024)
- [Bridging Local Details and Global Context in Text-Attributed Graphs](https://mlsanity.com/paper/2406.12608/) (2024)
- [HyperBERT: Mixing Hypergraph-Aware Layers with Language Models for Node Classification on Text-Attributed Hypergraphs](https://mlsanity.com/paper/2402.07309/) (2024)
- [Hypergraph Self-supervised Learning with Sampling-efficient Signals](https://mlsanity.com/paper/2404.11825/) (2024)
- [From Graphs to Hypergraphs: Hypergraph Projection and its Remediation](https://mlsanity.com/paper/2401.08519/) (2024)
- [Graph Contrastive Learning with Cohesive Subgraph Awareness](https://mlsanity.com/paper/2401.17580/) (2024)
- [High-Frequency-aware Hierarchical Contrastive Selective Coding for Representation Learning on Text-attributed Graphs](https://mlsanity.com/paper/2402.16240/) (2024)
- [How Transitive Are Real-World Group Interactions? -- Measurement and Reproduction](https://mlsanity.com/paper/2306.02358/) (2023)
- [HyGEN: Regularizing Negative Hyperedge Generation for Accurate Hyperedge Prediction](https://mlsanity.com/paper/2502.05827/) (2025)

## Built on by (most cited first, 2 of 2)

- [PRoH: Dynamic Planning and Reasoning over Knowledge Hypergraphs for Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.12434/) (2025)
- [HyperSU: Corpus-Driven Semantic-Unit Hypergraph for Retrieval-Augmented Generation](https://mlsanity.com/paper/2606.28351/) (2026)

## Abstract (arXiv)

Contrastive learning (CL) has become a dominant paradigm for self-supervised hypergraph learning, enabling effective training without costly labels. However, node entities in real-world hypergraphs are often associated with rich textual information, which has been largely ignored in prior works. Directly applying existing CL-based methods to such text-attributed hypergraphs (TAHGs) leads to three key limitations: (1) The common use of graph-agnostic text encoders fails to capture the correlations between textual semantics and hypergraph topology, resulting in less expressive representations. (2) Their reliance on random data augmentations introduces noise and weakens the contrastive signals. (3) The primary focus on node- and hyperedge-level contrastive signals limits the ability to capture long-range dependencies, which is essential for effective representation learning. To address these challenges, we introduce HiTeC, a two-stage hierarchical contrastive learning framework for effective self-supervised learning on TAHGs. In the first stage, we pre-train the text encoder with a structure-aware contrastive objective to overcome the graph-agnostic nature of conventional methods. In the second stage, we begin by introducing semantic-aware augmentations, including structure-contextualized text augmentation and semantic-aware hyperedge dropping, to facilitate informative view generation. Subsequently, we propose a multi-scale contrastive loss with an s-walk-based subgraph-level objective to capture long-range dependencies. Extensive experiments on six real-world datasets validate the effectiveness of our proposed method.