mlsanityData to 14 Oct 2026

When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming

arXiv 2306.04930 ↗First posted 8 Jun 2023cs.HC cs.LG cs.SEcited by 25

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

AI powered code-recommendation systems, such as Copilot and CodeWhisperer, provide code suggestions inside a programmer's environment (e.g., an IDE) with the aim of improving productivity. We pursue mechanisms for leveraging signals about programmers' acceptance and rejection of code suggestions to guide recommendations. We harness data drawn from interactions with GitHub Copilot, a system used by millions of programmers, to develop interventions that can save time for programmers. We introduce a utility-theoretic framework to drive decisions about suggestions to display versus withhold. The approach, conditional suggestion display from human feedback (CDHF), relies on a cascade of models that provide the likelihood that recommended code will be accepted. These likelihoods are used to selectively hide suggestions, reducing both latency and programmer verification time. Using data from 535 programmers, we perform a retrospective evaluation of CDHF and show that we can avoid displaying a significant fraction of suggestions that would have been rejected. We further demonstrate the importance of incorporating the programmer's latent unobserved state in decisions about when to display suggestions through an ablation study. Finally, we showcase how using suggestion acceptance as a reward signal for guiding the display of suggestions can lead to suggestions of reduced quality, indicating an unexpected pitfall.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 54 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

28 of 54 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  10. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  11. 2021Measuring Coding Challenge Competence With APPScited by 902
  12. 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
16 more
  1. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
  2. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 218
  3. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 158
  4. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 152
  5. 2019The Algorithmic Automation Problem: Prediction, Triage, and Human Effortcited by 101
  6. 2022Productivity Assessment of Neural Code Completioncited by 98
  7. 2022Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programmingcited by 91
  8. 2021Differentiable Learning Under Triagecited by 76
  9. 2023Who Should Predict? Exact Algorithms For Learning to Defer to Humanscited by 70
  10. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  11. 2022Sample Efficient Learning of Predictors that Complement Humanscited by 46
  12. 2021Perfection Not Required? Human-AI Partnerships in Code Translationcited by 42
  13. 2022What is it like to program with artificial intelligence?cited by 39
  14. 2021Teaching Humans When To Defer to a Classifier via Exemplarscited by 29
  15. 2023Learning Personalized Decision Support Policiescited by 14
  16. 2013Attention-Sensitive Alertingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 10; 2026: 6.

25 papers.

Sort:

By area:

  1. 2024"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trustcited by 67
  2. 2024Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forwardcited by 45
  3. 2025Navigating Rifts in Human-LLM Grounding: Study and Benchmarkcited by 29
  4. 2023Flows: Building Blocks of Reasoning and Collaborating AIcited by 25
  5. 2023The Rise of the AI Co-Pilot: Lessons for Design from Aviation and Beyondcited by 15
  6. 2024When Neural Code Completion Models Size up the Situation: Attaining Cheaper and Faster Completion through Dynamic Model Inferencecited by 10
  7. 2024Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scarycited by 10
  8. 2023Pitfalls in Language Models for Code Intelligence: A Taxonomy and Surveycited by 9
  9. 2025Explaining and Improving Information Complementarities in Multi-Agent Decision-makingcited by 6
  10. 2025Human-AI Experience in Integrated Development Environments: A Systematic Literature Reviewcited by 5
  11. 2023When Are Two Lists Better than One?: Benefits and Harms in Joint Decision-makingcited by 4
  12. 2025"My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistantscited by 4
13 more
  1. 2025Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomycited by 3
  2. 2026Multi-Round Human-AI Collaboration with User-Specified Requirementscited by 3
  3. 2026Agentic Coding Needs Proactivity, Not Just Autonomycited by 3
  4. 2024A Dynamic Model of Performative Human-ML Collaboration: Theory and Empirical Evidencecited by 2
  5. 2025CCCI: Code Completion with Contextual Information for Complex Data Transfer Tasks Using Large Language Modelscited by 2
  6. 2025Optimizing LLM Code Suggestions: Feedback-Driven Timing with Lightweight State Boundscited by 2
  7. 2025Pre-Filtering Code Suggestions using Developer Behavioral Telemetry to Optimize LLM-Assisted Programmingcited by 2
  8. 2026ComplLLM: Fine-tuning LLMs to Discover Complementary Signals for Decision-makingcited by 2
  9. 2025The Role of GitHub Copilot on Software Development: A Perspective on Productivity, Security, Best Practices and Future Directionscited by 1
  10. 2025Toolsuite for Implementing Multiagent Systems Based on Communication Protocolscited by 0
  11. 2026A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheetscited by 0
  12. 2026Propose to Learn, Learn to Propose: Evaluability-Aware Assistance under Bounded Rationalitycited by 0
  13. 2026ParallelPilot: Supporting Coordination and Monitoring in Parallel AI Codingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Human-AI Synergy in Agentic Code Review
  2. 2026From Code Review to Code Critique: Intent, Drift, and Spotlight for AI-Generated Diffs at Scale
  3. 2026Rethinking Code Review in the Age of AI: A Vision for Agentic Code Review
  4. 2025Beyond Autocomplete: Designing CopilotLens Towards Transparent and Explainable AI Coding Agents
  5. 2026From Human-Centric to Agentic Code Review: The Impact of Different Generations of Generative AI Technology on Review Quality
  6. 2026Predicting Acceptance and Review Effort in Human and Agent Pull Requests
  7. 2026These Aren't the Reviews You're Looking For How Humans Review AI-Generated Pull Requests
  8. 2026"Go Home Copilot, You're Drunk": Understanding Developer Responses to Agent-Generated Code Review Comments