mlsanityData to 7 Oct 2026

The Invisible Lottery: How Subtle Cues Steer Algorithm Choice in LLM Code Generation

arXiv 2606.04057 ↗First posted 2 Jun 2026cs.SE cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) now generate substantial production code, often for tasks with multiple valid algorithmic solutions. Incidental prompt cues, meaning contextual words or metadata outside the task specification, can steer which algorithm the model selects, even when all outputs pass the same tests. Prompt sensitivity is well studied as a tool to improve output quality. Here, output policy means algorithm choice under fixed correctness. We define algorithm steering as cue-induced shifts in algorithm-family distributions and run 46,535 controlled experiments across 11 tasks, 19 cue types (18 channels plus a memoization semantic-vs-surface ablation that preserves meaning while changing typography and punctuation), and 15 model configurations. We find large, systematic shifts in algorithm-family distributions (up to 100 pp), largely consistent with cue semantics, including in applied tasks such as rate limiting. Direct algorithm naming is the most reliable mitigation we tested. Accidental context therefore creates an "invisible lottery" over performance, security, and maintainability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 24 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

20 of 24 references.

References · 20

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  8. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  9. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  10. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  11. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 318
  12. 2018code2vec: Learning Distributed Representations of Codecited by 249
8 more
  1. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
  2. 2022Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programmingcited by 91
  3. 2023When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Modelscited by 50
  4. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 33
  5. 2024LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completioncited by 29
  6. 2024Selective Prompt Anchoring for Code Generationcited by 16
  7. 2025CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoningcited by 14
  8. 2025How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Codecited by 9

Built on by

Not cited yet.