# Towards a Holistic Understanding of Mathematical Questions with Contrastive Pre-training

arXiv: https://arxiv.org/abs/2301.07558 · HTML: https://mlsanity.com/paper/2301.07558/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 18 Jan 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [EKT: Exercise-aware Knowledge Tracing for Student Performance Prediction](https://mlsanity.com/paper/1906.05658/) (2019)
- [MathBERT: A Pre-Trained Model for Mathematical Formula Understanding](https://mlsanity.com/paper/2105.00377/) (2021)
- [Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classification](https://mlsanity.com/paper/2203.03825/) (2022)
- [Improved Text Classification via Contrastive Adversarial Training](https://mlsanity.com/paper/2107.10137/) (2021)
- [Ranking Info Noise Contrastive Estimation: Boosting Contrastive Learning via Ranked Positives](https://mlsanity.com/paper/2201.11736/) (2022)
- [Sequence Level Contrastive Learning for Text Summarization](https://mlsanity.com/paper/2109.03481/) (2021)
- [Acquiring Knowledge from Pre-trained Model to Neural Machine Translation](https://mlsanity.com/paper/1912.01774/) (2019)
- [LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding](https://mlsanity.com/paper/2012.07335/) (2020)
- [QuesNet: A Unified Representation for Heterogeneous Test Questions](https://mlsanity.com/paper/1905.10949/) (2019)
- [Context-Guided BERT for Targeted Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2010.07523/) (2020)
- [R2DE: a NLP approach to estimating IRT parameters of newly generated questions](https://mlsanity.com/paper/2001.07569/) (2020)
- [JiuZhang: A Chinese Pre-trained Language Model for Mathematical Problem Understanding](https://mlsanity.com/paper/2206.06315/) (2022)
- [Introducing a framework to assess newly created questions with Natural Language Processing](https://mlsanity.com/paper/2004.13530/) (2020)

## Built on by (most cited first, 5 of 5)

- [A Novel Approach for Auto-Formulation of Optimization Problems](https://mlsanity.com/paper/2302.04643/) (2023)
- [Survey of Computerized Adaptive Testing: A Machine Learning Perspective](https://mlsanity.com/paper/2404.00712/) (2024)
- [GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Models](https://mlsanity.com/paper/2511.11134/) (2025)
- [Leveraging Label Semantics and Meta-Label Refinement for Multi-Label Question Classification](https://mlsanity.com/paper/2411.01841/) (2024)
- [EduNLP: Towards a Unified and Modularized Library for Educational Resources](https://mlsanity.com/paper/2406.01276/) (2024)

## Abstract (arXiv)

Understanding mathematical questions effectively is a crucial task, which can benefit many applications, such as difficulty estimation. Researchers have drawn much attention to designing pre-training models for question representations due to the scarcity of human annotations (e.g., labeling difficulty). However, unlike general free-format texts (e.g., user comments), mathematical questions are generally designed with explicit purposes and mathematical logic, and usually consist of more complex content, such as formulas, and related mathematical knowledge (e.g., Function). Therefore, the problem of holistically representing mathematical questions remains underexplored. To this end, in this paper, we propose a novel contrastive pre-training approach for mathematical question representations, namely QuesCo, which attempts to bring questions with more similar purposes closer. Specifically, we first design two-level question augmentations, including content-level and structure-level, which generate literally diverse question pairs with similar purposes. Then, to fully exploit hierarchical information of knowledge concepts, we propose a knowledge hierarchy-aware rank strategy (KHAR), which ranks the similarities between questions in a fine-grained manner. Next, we adopt a ranking contrastive learning task to optimize our model based on the augmented and ranked questions. We conduct extensive experiments on two real-world mathematical datasets. The experimental results demonstrate the effectiveness of our model.