# Multi-Level Attention and Contrastive Learning for Enhanced Text Classification with an Optimized Transformer

arXiv: https://arxiv.org/abs/2501.13467 · HTML: https://mlsanity.com/paper/2501.13467/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 23 Jan 2025.

## Builds on (most cited first, 12 of 12)

- [Transforming Multidimensional Time Series into Interpretable Event Sequences for Advanced Data Mining](https://mlsanity.com/paper/2409.14327/) (2024)
- [Self-Supervised Graph Neural Networks for Enhanced Feature Extraction in Heterogeneous Information Networks](https://mlsanity.com/paper/2410.17617/) (2024)
- [Leveraging Convolutional Neural Network-Transformer Synergy for Predictive Modeling in Risk-Based Applications](https://mlsanity.com/paper/2412.18222/) (2024)
- [Tree-based RAG-Agent Recommendation System: A Case Study in Medical Test Data](https://mlsanity.com/paper/2501.02727/) (2025)
- [Machine Learning Techniques for Pattern Recognition in High-Dimensional Data Mining](https://mlsanity.com/paper/2412.15593/) (2024)
- [Accurate Medical Named Entity Recognition Through Specialized NLP Models](https://mlsanity.com/paper/2412.08255/) (2024)
- [Dynamic Scheduling Strategies for Resource Optimization in Computing Environments](https://mlsanity.com/paper/2412.17301/) (2024)
- [Collaborative Optimization in Financial Data Mining Through Deep Learning and ResNeXt](https://mlsanity.com/paper/2412.17314/) (2024)
- [A Matrix Logic Approach to Efficient Frequent Itemset Discovery in Large Data Sets](https://mlsanity.com/paper/2412.19420/) (2024)
- [Adaptive User Interface Generation Through Reinforcement Learning: A Data-Driven Approach to Personalization and Optimization](https://mlsanity.com/paper/2412.16837/) (2024)
- [Time Series Feature Redundancy Paradox: An Empirical Study Based on Mortgage Default Prediction](https://mlsanity.com/paper/2501.00034/) (2024)
- [Computer Vision-Driven Gesture Recognition: Toward Natural and Intuitive Human-Computer](https://mlsanity.com/paper/2412.18321/) (2024)

## Built on by (most cited first, 5 of 5)

- [Graph Neural Network-Driven Hierarchical Mining for Complex Imbalanced Data](https://mlsanity.com/paper/2502.03803/) (2025)
- [Optimized Unet with Attention Mechanism for Multi-Scale Semantic Segmentation](https://mlsanity.com/paper/2502.03813/) (2025)
- [Context-Aware Rule Mining Using a Dynamic Transformer-Based Framework](https://mlsanity.com/paper/2503.11125/) (2025)
- [Adaptive Transformer Attention and Multi-Scale Fusion for Spine 3D Segmentation](https://mlsanity.com/paper/2503.12853/) (2025)
- [PerSoMed: A Large-Scale Balanced Dataset for Persian Social Media Text Classification](https://mlsanity.com/paper/2602.19333/) (2026)

## Abstract (arXiv)

This paper studies a text classification algorithm based on an improved Transformer to improve the performance and efficiency of the model in text classification tasks. Aiming at the shortcomings of the traditional Transformer model in capturing deep semantic relationships and optimizing computational complexity, this paper introduces a multi-level attention mechanism and a contrastive learning strategy. The multi-level attention mechanism effectively models the global semantics and local features in the text by combining global attention with local attention; the contrastive learning strategy enhances the model's ability to distinguish between different categories by constructing positive and negative sample pairs while improving the classification effect. In addition, in order to improve the training and inference efficiency of the model on large-scale text data, this paper designs a lightweight module to optimize the feature transformation process and reduce the computational cost. Experimental results on the dataset show that the improved Transformer model outperforms the comparative models such as BiLSTM, CNN, standard Transformer, and BERT in terms of classification accuracy, F1 score, and recall rate, showing stronger semantic representation ability and generalization performance. The method proposed in this paper provides a new idea for algorithm optimization in the field of text classification and has good application potential and practical value. Future work will focus on studying the performance of this model in multi-category imbalanced datasets and cross-domain tasks and explore the integration wi