mlsanityData to 7 Oct 2026

Selective Prompt Anchoring for Code Generation

arXiv 2408.09121 ↗First posted 17 Aug 2024cs.LG cs.AI cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Recent advances in large language models (LLMs) have transformed software development by automatically generating code from natural language. Yet challenges remain in generating fully correct code that aligns with user intent. Our study reveals that LLMs tend to pay less attention to user prompts as more code tokens are generated. We hypothesize that this attention dilution issue is an important reason for code generation errors. To mitigate this issue, we propose Selective Prompt Anchoring (SPA) to guide code LLMs to pay more attention to user intent when generating code. We evaluate SPA using six base LLMs across six benchmarks. Our results demonstrate that SPA enhances Pass@1 by up to 12.9%, consistently outperforming SOTA code generation methods in all settings. Our code is available at https://github.com/magic-YuanTian/Selective-Prompt-Anchoring.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 73 of its 95 references that are in the map. First posted on arXiv 17 Aug 2024.

Builds on

73 of 95 references.

References · 73

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2021Program Synthesis with Large Language Modelscited by 3,509
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
38 more
  1. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  6. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  9. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  10. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  11. 2023Teaching Large Language Models to Self-Debugcited by 963
  12. 2021Measuring Coding Challenge Competence With APPScited by 879
  13. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  14. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  15. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  16. 2023StarCoder: may the source be with you!cited by 790
  17. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  18. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  19. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  20. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  21. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  22. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  23. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  24. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  25. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  26. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  27. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  28. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  29. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  30. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  31. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  32. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  33. 2016Understanding Neural Networks through Representation Erasurecited by 223
  34. 2019SPoC: Search-based Pseudocode to Codecited by 176
  35. 2016Language to Logical Form with Neural Attentioncited by 158
  36. 2023Self-planning Code Generation with Large Language Modelscited by 121
  37. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 110
  38. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 12.

16 papers.

Sort:

By area:

  1. 2025RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Codecited by 28
  2. 2026TrajAudit: Automated Failure Diagnosis for Agentic Coding Systemscited by 11
  3. 2025Text-to-SQL Domain Adaptation via Human-LLM Collaborative Data Annotationcited by 6
  4. 2026Spectral Attention Steering for Prompt Highlightingcited by 6
  5. 2026PV-SQL: Synergizing Database Probing and Rule-based Verification for Text-to-SQL Agentscited by 5
  6. 2025Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environmentscited by 3
  7. 2026PRISM-Delta: Differential Subspace Steering for Prompt Highlighting in Large Language Modelscited by 2
  8. 2025Domain-Adapted Pre-trained Language Models for Implicit Information Extraction in Crash Narrativescited by 1
  9. 2026Autoregressive, Yet Revisable: In Decoding Revision for Secure Code Generationcited by 1
  10. 2026IntentCoding: Amplifying User Intent in Code Generationcited by 0
  11. 2026From Instruction to Output: The Role of Prompting in Modern NLGcited by 0
  12. 2026Enhancing Multi-Agent Communication through Attention Steering with Context Relevancecited by 0
4 more
  1. 2026The Invisible Lottery: How Subtle Cues Steer Algorithm Choice in LLM Code Generationcited by 0
  2. 2026ISEE: Interactive Semantic Enrichment for Database Fieldscited by 0
  3. 2026Training-Free Token-Level Steering for LLM Personalized Co-Writingcited by 0
  4. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0