# Class-level Structural Relation Modelling and Smoothing for Visual Representation Learning

arXiv: https://arxiv.org/abs/2308.04142 · HTML: https://mlsanity.com/paper/2308.04142/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 61 references that are in the map.
- First posted on arXiv 8 Aug 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Few-Shot Learning with Graph Neural Networks](https://mlsanity.com/paper/1711.04043/) (2017)
- [Generalizing Across Domains via Cross-Gradient Training](https://mlsanity.com/paper/1804.10745/) (2018)
- [Cross Attention Network for Few-shot Classification](https://mlsanity.com/paper/1910.07677/) (2019)
- [Balanced Contrastive Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2207.09052/) (2022)
- [Visual Compositional Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2007.12407/) (2020)
- [Deconfounded Video Moment Retrieval with Causal Intervention](https://mlsanity.com/paper/2106.01534/) (2021)
- [Curricular Contrastive Regularization for Physics-aware Single Image Dehazing](https://mlsanity.com/paper/2303.14218/) (2023)
- [Embedding Propagation: Smoother Manifold for Few-Shot Classification](https://mlsanity.com/paper/2003.04151/) (2020)
- [On the Integration of Self-Attention and Convolution](https://mlsanity.com/paper/2111.14556/) (2021)
- [Tree-Augmented Cross-Modal Encoding for Complex-Query Video Retrieval](https://mlsanity.com/paper/2007.02503/) (2020)
- [Learning to Resize Images for Computer Vision Tasks](https://mlsanity.com/paper/2103.09950/) (2021)
- [CUDA: Curriculum of Data Augmentation for Long-Tailed Recognition](https://mlsanity.com/paper/2302.05499/) (2023)
- [BatchFormer: Learning to Explore Sample Relationships for Robust Representation Learning](https://mlsanity.com/paper/2203.01522/) (2022)
- [Constructing Balance from Imbalance for Long-tailed Image Recognition](https://mlsanity.com/paper/2208.02567/) (2022)
- [ProCo: Prototype-aware Contrastive Learning for Long-tailed Medical Image Classification](https://mlsanity.com/paper/2209.00183/) (2022)
- [Triple Sequence Learning for Cross-domain Recommendation](https://mlsanity.com/paper/2304.05027/) (2023)
- [Incomplete Multi-View Multi-Label Learning via Label-Guided Masked View- and Category-Aware Transformers](https://mlsanity.com/paper/2303.07180/) (2023)
- [Meta-Causal Feature Learning for Out-of-Distribution Generalization](https://mlsanity.com/paper/2208.10156/) (2022)
- [Cross-Modal Content Inference and Feature Enrichment for Cold-Start Recommendation](https://mlsanity.com/paper/2307.02761/) (2023)
- [Affinity Graph Supervision for Visual Recognition](https://mlsanity.com/paper/2003.09049/) (2020)

## Built on by (most cited first, 9 of 9)

- [Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language Models for Domain-Generalized Semantic Segmentation](https://mlsanity.com/paper/2504.03193/) (2025)
- [Class-wise Balancing Data Replay for Federated Class-Incremental Learning](https://mlsanity.com/paper/2507.07712/) (2025)
- [3DOT: Texture Transfer for 3DGS Objects from a Single Reference Image](https://mlsanity.com/paper/2503.18853/) (2025)
- [GSAlign: Geometric and Semantic Alignment Network for Aerial-Ground Person Re-Identification](https://mlsanity.com/paper/2510.22268/) (2025)
- [Global Intervention and Distillation for Federated Out-of-Distribution Generalization](https://mlsanity.com/paper/2504.00850/) (2025)
- [ProtoConNet: Prototypical Augmentation and Alignment for Open-Set Few-Shot Image Classification](https://mlsanity.com/paper/2507.11845/) (2025)
- [Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal Alignment](https://mlsanity.com/paper/2407.18854/) (2024)
- [Comparative Study of Neighbor-based Methods for Local Outlier Detection](https://mlsanity.com/paper/2405.19247/) (2024)
- [From Selection to Scheduling: Federated Geometry-Aware Correction Makes Exemplar Replay Work Better under Continual Dynamic Heterogeneity](https://mlsanity.com/paper/2604.08617/) (2026)

## Abstract (arXiv)

Representation learning for images has been advanced by recent progress in more complex neural models such as the Vision Transformers and new learning theories such as the structural causal models. However, these models mainly rely on the classification loss to implicitly regularize the class-level data distributions, and they may face difficulties when handling classes with diverse visual patterns. We argue that the incorporation of the structural information between data samples may improve this situation. To achieve this goal, this paper presents a framework termed Class-level Structural Relation Modeling and Smoothing for Visual Representation Learning (CSRMS), which includes the Class-level Relation Modelling, Class-aware Graph Sampling, and Relational Graph-Guided Representation Learning modules to model a relational graph of the entire dataset and perform class-aware smoothing and regularization operations to alleviate the issue of intra-class visual diversity and inter-class similarity. Specifically, the Class-level Relation Modelling module uses a clustering algorithm to learn the data distributions in the feature space and identify three types of class-level sample relations for the training set; Class-aware Graph Sampling module extends typical training batch construction process with three strategies to sample dataset-level sub-graphs; and Relational Graph-Guided Representation Learning module employs a graph convolution network with knowledge-guided smoothing operations to ease the projection from different visual patterns to the same class. Experiments demonstrate the effectiveness of structured knowledge modelling for enhanced representation learning and show that CSRMS can be incorporated with any state-of-the-art visual representation learning models for performance gains. The source codes and demos have been released at https://github.com/czt117/CSRMS.