mlsanityData to 14 Oct 2026

Correct Choices, Failed Sequences: Choice2Seq Benchmarks Correctness Retention Across Output Interfaces

arXiv 2610.16191 ↗First posted 12 Oct 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

A model may select the correct answer yet fail to construct it through a sequence of local decisions. We introduce Choice2Seq, a diagnostic benchmark that measures correctness retention across complete-answer selection and sequential construction in independent runs on the same problems and candidate sets. Sequence Composition Rate (SCR) measures the fraction of correctly selected items that are also answered correctly through local construction. Jev achieves 97.1% and 99.5% selection accuracy on arithmetic and lexical retrieval, yet retains only 33.0% and 2.1% of those correct choices under serial construction, respectively. Substantial gaps also emerge for StartLux and Qwen, including when Qwen emits characters directly. Restricting local options to candidate-supported characters improves construction, while explicit position indexing improves copying fidelity, even for incorrect selections. These interventions distinguish task correctness, plan fidelity, and output compliance. Matched dialogue-state evaluations reveal that local field construction largely preserves Jev's correct states but reduces Qwen's mean-turn accuracy, highlighting task- and interface-dependent differences. Sequence-level confidence and request-cost analyses further characterize completed outputs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 39 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

19 of 39 references.

References · 19

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  3. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  4. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  5. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 434
  6. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 108
  7. 2024Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelscited by 96
  8. 2025JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Modelscited by 71
  9. 2019Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?cited by 35
  10. 2024CUTE: Measuring LLMs' Understanding of Their Tokenscited by 31
  11. 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 30
  12. 2021Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokenscited by 27
7 more
  1. 2026Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to Itcited by 25
  2. 2026Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understandingcited by 11
  3. 2026Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?cited by 6
  4. 2025Spelling-out is not Straightforward: LLMs' Capability of Tokenization from Token to Characterscited by 4
  5. 2025Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracycited by 3
  6. 2026Offline Preference-Based Trajectory Evaluationcited by 2
  7. 2026SecJev: Bringing Security Expertise to System One Decision Modelscited by 1

Built on by

Not cited yet.