mlsanityData to 14 Oct 2026

BoN Appetit Team at LeWiDi-2025: Best-of-N Test-time Scaling Can Not Stomach Annotation Disagreements (Yet)

arXiv 2510.12516 ↗First posted 14 Oct 2025cs.CL cs.AIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Test-time scaling is a family of techniques to improve LLM outputs at inference time by performing extra computation. To the best of our knowledge, test-time scaling has been limited to domains with verifiably correct answers, like mathematics and coding. We transfer test-time scaling to the LeWiDi-2025 tasks to evaluate annotation disagreements. We experiment with three test-time scaling methods: two benchmark algorithms (Model Averaging and Majority Voting), and a Best-of-N sampling method. The two benchmark methods improve LLM performance consistently on the LeWiDi tasks, but the Best-of-N method does not. Our experiments suggest that the Best-of-N method does not currently transfer from mathematics to LeWiDi tasks, and we analyze potential reasons for this gap.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 5 of its 42 references that are in the map. First posted on arXiv 14 Oct 2025.

Builds on

5 of 42 references.

References · 5

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
  3. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling
  4. 2025The Art of Scaling Test-Time Compute for Large Language Models
  5. 2025Test-time Scaling of LLMs: A Survey from A Subproblem Structure Perspective
  6. 2025Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?
  7. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters