# Diet Code Is Healthy: Simplifying Programs for Pre-trained Models of Code

arXiv: https://arxiv.org/abs/2206.14390 · HTML: https://mlsanity.com/paper/2206.14390/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2022.

## Built on by (most cited first, 24 of 24)

- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [Reducing Cost of LLM Agents with Trajectory Reduction](https://mlsanity.com/paper/2509.23586/) (2025)
- [SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents](https://mlsanity.com/paper/2601.16746/) (2026)
- [LongCodeZip: Compress Long Context for Code Language Models](https://mlsanity.com/paper/2510.00446/) (2025)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [Do Machines and Humans Focus on Similar Code? Exploring Explainability of Large Language Models in Code Summarization](https://mlsanity.com/paper/2402.14182/) (2024)
- [Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?](https://mlsanity.com/paper/2306.01220/) (2023)
- [Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understanding](https://mlsanity.com/paper/2602.01785/) (2026)
- [The Hidden Cost of Readability: How Code Formatting Silently Consumes Your LLM Budget](https://mlsanity.com/paper/2508.13666/) (2025)
- [Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey](https://mlsanity.com/paper/2310.17903/) (2023)
- [Can Vision-Language Models Handle Long-Context Code? An Empirical Study on Visual Compression](https://mlsanity.com/paper/2602.00746/) (2026)
- [A Critical Study of What Code-LLMs (Do Not) Learn](https://mlsanity.com/paper/2406.11930/) (2024)
- [Study of Distractors in Neural Models of Code](https://mlsanity.com/paper/2303.01739/) (2023)
- [Trojans in Large Language Models of Code: A Critical Review through a Trigger-Based Taxonomy](https://mlsanity.com/paper/2405.02828/) (2024)
- [Analyzing Latent Concepts in Code Language Models](https://mlsanity.com/paper/2510.00476/) (2025)
- [SWE-Pruner Pro: The Coder LLM Already Knows What to Prune](https://mlsanity.com/paper/2607.18213/) (2026)
- [Redundancy and Concept Analysis for Code-trained Language Models](https://mlsanity.com/paper/2305.00875/) (2023)
- [Beyond Self-learned Attention: Mitigating Attention Bias in Transformer-based Models Using Attention Guidance](https://mlsanity.com/paper/2402.16790/) (2024)
- [CONCORD: Towards a DSL for Configurable Graph Code Representation](https://mlsanity.com/paper/2401.17967/) (2024)
- [LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Models](https://mlsanity.com/paper/2505.14759/) (2025)
- [Repairing Regex Vulnerabilities via Localization-Guided Instructions](https://mlsanity.com/paper/2510.09037/) (2025)
- [When Compression Becomes an Attack Surface: Black-Box Attacks on Prompt-Compressed LLM Agents](https://mlsanity.com/paper/2510.22963/) (2025)
- [From User Interface to Agent Interface: Efficiency Optimization of UI Representations for LLM Agents](https://mlsanity.com/paper/2512.13438/) (2025)
- [CodeShrink: Adaptive Visual Compression for Efficient Multimodal Code Understanding](https://mlsanity.com/paper/2607.29637/) (2026)

## Abstract (arXiv)

Pre-trained code representation models such as CodeBERT have demonstrated superior performance in a variety of software engineering tasks, yet they are often heavy in complexity, quadratically with the length of the input sequence. Our empirical analysis of CodeBERT's attention reveals that CodeBERT pays more attention to certain types of tokens and statements such as keywords and data-relevant statements. Based on these findings, we propose DietCode, which aims at lightweight leverage of large pre-trained models for source code. DietCode simplifies the input program of CodeBERT with three strategies, namely, word dropout, frequency filtering, and an attention-based strategy which selects statements and tokens that receive the most attention weights during pre-training. Hence, it gives a substantial reduction in the computational cost without hampering the model performance. Experimental results on two downstream tasks show that DietCodeBERT provides comparable results to CodeBERT with 40% less computational cost in fine-tuning and testing.