mlsanityData to 14 Oct 2026

Diet Code Is Healthy: Simplifying Programs for Pre-trained Models of Code

arXiv 2206.14390 ↗First posted 29 Jun 2022cs.SEcited by 24

Not on the map. Why?

Abstract (arXiv)

Pre-trained code representation models such as CodeBERT have demonstrated superior performance in a variety of software engineering tasks, yet they are often heavy in complexity, quadratically with the length of the input sequence. Our empirical analysis of CodeBERT's attention reveals that CodeBERT pays more attention to certain types of tokens and statements such as keywords and data-relevant statements. Based on these findings, we propose DietCode, which aims at lightweight leverage of large pre-trained models for source code. DietCode simplifies the input program of CodeBERT with three strategies, namely, word dropout, frequency filtering, and an attention-based strategy which selects statements and tokens that receive the most attention weights during pre-training. Hence, it gives a substantial reduction in the computational cost without hampering the model performance. Experimental results on two downstream tasks show that DietCodeBERT provides comparable results to CodeBERT with 40% less computational cost in fine-tuning and testing.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 Jun 2022.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 5; 2025: 8; 2026: 5.

24 papers.

Sort:

By area:

  1. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  2. 2025Reducing Cost of LLM Agents with Trajectory Reductioncited by 48
  3. 2026SWE-Pruner: Self-Adaptive Context Pruning for Coding Agentscited by 44
  4. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 39
  5. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
  6. 2024Do Machines and Humans Focus on Similar Code? Exploring Explainability of Large Language Models in Code Summarizationcited by 16
  7. 2023Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?cited by 14
  8. 2026Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understandingcited by 13
  9. 2025The Hidden Cost of Readability: How Code Formatting Silently Consumes Your LLM Budgetcited by 12
  10. 2023Pitfalls in Language Models for Code Intelligence: A Taxonomy and Surveycited by 9
  11. 2026Can Vision-Language Models Handle Long-Context Code? An Empirical Study on Visual Compressioncited by 5
  12. 2024A Critical Study of What Code-LLMs (Do Not) Learncited by 4
12 more
  1. 2023Study of Distractors in Neural Models of Codecited by 2
  2. 2024Trojans in Large Language Models of Code: A Critical Review through a Trigger-Based Taxonomycited by 2
  3. 2025Analyzing Latent Concepts in Code Language Modelscited by 2
  4. 2026SWE-Pruner Pro: The Coder LLM Already Knows What to Prunecited by 2
  5. 2023Redundancy and Concept Analysis for Code-trained Language Modelscited by 1
  6. 2024Beyond Self-learned Attention: Mitigating Attention Bias in Transformer-based Models Using Attention Guidancecited by 1
  7. 2024CONCORD: Towards a DSL for Configurable Graph Code Representationcited by 0
  8. 2025LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Modelscited by 0
  9. 2025Repairing Regex Vulnerabilities via Localization-Guided Instructionscited by 0
  10. 2025When Compression Becomes an Attack Surface: Black-Box Attacks on Prompt-Compressed LLM Agentscited by 0
  11. 2025From User Interface to Agent Interface: Efficiency Optimization of UI Representations for LLM Agentscited by 0
  12. 2026CodeShrink: Adaptive Visual Compression for Efficient Multimodal Code Understandingcited by 0