# Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large Language Models

arXiv: https://arxiv.org/abs/2405.11196 · HTML: https://mlsanity.com/paper/2405.11196/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2024.

## Built on by (most cited first, 10 of 10)

- [Reducing Cost of LLM Agents with Trajectory Reduction](https://mlsanity.com/paper/2509.23586/) (2025)
- [SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents](https://mlsanity.com/paper/2601.16746/) (2026)
- [LongCodeZip: Compress Long Context for Code Language Models](https://mlsanity.com/paper/2510.00446/) (2025)
- [Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understanding](https://mlsanity.com/paper/2602.01785/) (2026)
- [SWE-Pruner Pro: The Coder LLM Already Knows What to Prune](https://mlsanity.com/paper/2607.18213/) (2026)
- [Prompt-Driven Code Summarization: A Systematic Literature Review](https://mlsanity.com/paper/2604.15385/) (2026)
- [On Problems of Implicit Context Compression for Software Engineering Agents](https://mlsanity.com/paper/2605.11051/) (2026)
- [CodeShrink: Adaptive Visual Compression for Efficient Multimodal Code Understanding](https://mlsanity.com/paper/2607.29637/) (2026)
- [CIT-CAD: Constraint Intent Tree-based CAD Code Generation and Verification](https://mlsanity.com/paper/2609.07434/) (2026)
- [Compress What You See, Not What You Say: Anchored Context Distillation for Latent-Observation Software Engineering Agents](https://mlsanity.com/paper/2609.31430/) (2026)

## Abstract (arXiv)

Pre-trained Large Language Models (LLM) have achieved remarkable successes in several domains. However, code-oriented LLMs are heavy in computational complexity, and quadratically with the length of the input. Toward simplifying the input program of an LLM, the state-of-the-art approach has the strategies to filter the input code tokens based on the attention scores given by the LLM. The decision to simplify the input should not rely on the attention patterns of an LLM, as these patterns are influenced by both the model architecture and the pre-training dataset. Since the model and dataset are part of the solution domain, not the problem domain where the input belongs, the outcome may differ when the model is pre-trained on a different dataset. We propose SlimCode, a model-agnostic code simplification solution for LLMs that depends on the nature of input code tokens. As an empirical study on the LLMs including CodeBERT, CodeT5, and GPT-4 for two main tasks: code search and summarization, we reported that 1) the removal ratio of code has a linear-like relation with the saving ratio on training time, 2) the impact of categorized tokens on code simplification can vary significantly, 3) the impact of categorized tokens on code simplification is task-specific but model-agnostic, and 4) the above findings hold for the paradigm-prompt engineering and interactive in-context learning. The empirical results showed that SlimCode can improve the state-of-the-art technique by 9.46% and 5.15% in terms of MRR and BLEU score on code search and summarization. Moreover, SlimCode is 133 times faster than the state-of-the-art approach. Additionally, SlimCode can reduce the cost of invoking GPT-4 by up to 24% per API query, while still producing comparable results to those with the original code.