# Scaling Laws Behind Code Understanding Model

arXiv: https://arxiv.org/abs/2402.12813 · HTML: https://mlsanity.com/paper/2402.12813/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2024.

## Built on by (most cited first, 4 of 4)

- [On the Limitations of Compute Thresholds as a Governance Strategy](https://mlsanity.com/paper/2407.05694/) (2024)
- [Towards Better Code Understanding in Decoder-Only Models with Contrastive Learning](https://mlsanity.com/paper/2406.12326/) (2024)
- [Scaling Laws for Code: A More Data-Hungry Regime](https://mlsanity.com/paper/2510.08702/) (2025)
- [Standing on the Shoulders of Giants: Stabilized Knowledge Distillation for Cross--Language Code Clone Detection](https://mlsanity.com/paper/2605.02860/) (2026)

## Abstract (arXiv)

The scaling law is becoming a fundamental law in many machine learning areas. That is, test error falls off with the power law when increasing training data, model size, and computing resource. However, whether this law is suitable for the task of code understanding is not well studied, and most current language models for code understanding are about 100M parameters, which are relatively "small" compared to large language models. In this paper, we conduct extensive experiments to investigate the scaling law for the code understanding task by varying training data, model size, and computing resource. We validate that the test error of code understanding models falls off with the power law when using larger models, indicating that the scaling law is suitable for the code understanding task. Besides, we apply different scales of models to two downstream code understanding tasks, and find that the performance increases with larger scale of models. Finally, we train a large-scale code understanding model named CoLSBERT with 1.5B parameters on a large dataset using more computing resource, which outperforms previous work by a large margin. We will release our code and the CoLSBERT model when our paper is published.