# Learning in the Wild: Towards Leveraging Unlabeled Data for Effectively Tuning Pre-trained Code Models

arXiv: https://arxiv.org/abs/2401.01060 · HTML: https://mlsanity.com/paper/2401.01060/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jan 2024.

## Built on by (most cited first, 9 of 9)

- [Automatic Programming: Large Language Models and Beyond](https://mlsanity.com/paper/2405.02213/) (2024)
- [Source Code Summarization in the Era of Large Language Models](https://mlsanity.com/paper/2407.07959/) (2024)
- [Search-Based LLMs for Code Optimization](https://mlsanity.com/paper/2408.12159/) (2024)
- [How secure is AI-generated Code: A Large-Scale Comparison of Large Language Models](https://mlsanity.com/paper/2404.18353/) (2024)
- [How to Use Graph Data in the Wild to Help Graph Anomaly Detection?](https://mlsanity.com/paper/2506.04190/) (2025)
- [TREAT: A Code LLMs Trustworthiness / Reliability Evaluation and Testing Framework](https://mlsanity.com/paper/2510.17163/) (2025)
- [ReDef: Do Code Language Models Truly Understand Code Changes for Just-in-Time Software Defect Prediction?](https://mlsanity.com/paper/2509.09192/) (2025)
- [BatCoder: Self-Supervised Bidirectional Code-Documentation Learning via Back-Translation](https://mlsanity.com/paper/2602.02554/) (2026)
- [Human-Machine Co-Boosted Bug Report Identification with Mutualistic Neural Active Learning](https://mlsanity.com/paper/2604.18862/) (2026)

## Abstract (arXiv)

Pre-trained code models have recently achieved substantial improvements in many code intelligence tasks. These models are first pre-trained on large-scale unlabeled datasets in a task-agnostic manner using self-supervised learning, and then fine-tuned on labeled datasets in downstream tasks. However, the labeled datasets are usually limited in size (i.e., human intensive efforts), which may hinder the performance of pre-trained code models in specific tasks. To mitigate this, one possible solution is to leverage the large-scale unlabeled data in the tuning stage by pseudo-labeling. However, directly employing the pseudo-labeled data can bring a large amount of noise, i.e., incorrect labels, leading to suboptimal performance. How to effectively leverage the noisy pseudo-labeled data is a challenging yet under-explored problem.In this paper, we propose a novel approach named HINT to improve pre-trained code models with large-scale unlabeled datasets by better utilizing the pseudo-labeled data. HINT includes two main modules: HybrId pseudo-labeled data selection and Noise-tolerant Training. In the hybrid pseudo-data selection module, considering the robustness issue, apart from directly measuring the quality of pseudo labels through training loss, we further propose to employ a retrieval-based method to filter low-quality pseudo-labeled data. The noise-tolerant training module aims to further mitigate the influence of errors in pseudo labels by training the model with a noise-tolerant loss function and by regularizing the consistency of model predictions.The experimental results show that HINT can better leverage those unlabeled data in a task-specific way and provide complementary benefits for pre-trained models, e.g., improving the best baseline model by 15.33%, 16.50%, and 8.98% on code summarization, defect detection, and assertion generation, respectively.