mlsanityData to 7 Oct 2026

The Perplexity Paradox: Why Code Compresses Better Than Math in LLM Prompts

arXiv 2602.15843 ↗First posted 21 Jan 2026cs.CL cs.AIcited by 2

On the map: LLM reasoning → Mathematical reasoning

also in 1 more
Abstract (arXiv)

In "Compress or Route?" (Johnson, 2026), we found that code generation tolerates aggressive prompt compression (r >= 0.6) while chain-of-thought reasoning degrades gradually. That study was limited to HumanEval (164 problems), left the "perplexity paradox" mechanism unvalidated, and provided no adaptive algorithm. This paper addresses all three gaps. First, we validate across six code benchmarks (HumanEval, MBPP, HumanEval+, MultiPL-E) and four reasoning benchmarks (GSM8K, MATH, ARC-Challenge, MMLU-STEM), confirming the compression threshold generalizes across languages and difficulties. Second, we conduct the first per-token perplexity analysis (n=723 tokens), revealing a "perplexity paradox": code syntax tokens are preserved (high perplexity) while numerical values in math problems are pruned despite being task-critical (low perplexity). Signature injection recovers +34 percentage points in pass rate (5.3% to 39.3%; Cohen's h=0.890). Third, we propose TAAC (Task-Aware Adaptive Compression), achieving 22% cost reduction with 96% quality preservation, outperforming fixed-ratio compression by 7%. MBPP validation (n=1,800 trials) confirms systematic variation: 3.6% at r=0.3 to 54.6% at r=1.0.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 52 references that are in the map. First posted on arXiv 21 Jan 2026.

Builds on

40 of 52 references.

References · 40

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
28 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  9. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  11. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  12. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  13. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  14. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  15. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  16. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  17. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  18. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  19. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  20. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  21. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  22. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  23. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  24. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  25. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  26. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 96
  27. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  28. 2024TACO-RL: Task Aware Prompt Compression Optimization with Reinforcement Learningcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Prompt Compression in Production Task Orchestration: A Pre-Registered Randomized Trialcited by 1
  2. 2026The Compression Paradox in LLM Inference: Provider-Dependent Energy Effects of Prompt Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025How Well do LLMs Compress Their Own Chain-of-Thought? A Token Complexity Approach
  2. 2025On Code-Induced Reasoning in LLMs
  3. 2025CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning
  4. 2025Chain-of-Code Collapse: Reasoning Failures in LLMs via Adversarial Prompting in Code Generation
  5. 2026Less Is More: Cognitive Load and the Single-Prompt Ceiling in LLM Mathematical Reasoning
  6. 2025Exploring LLM Reasoning Through Controlled Prompt Variations
  7. 2024Large Language Models Struggle with Unreasonability in Math Problems
  8. 2026Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations