When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering
Not on the map. Why?
Abstract (arXiv)
A model's agreement across perturbed inputs is used both as a label-free reliability signal and as a self-training target, on the premise that agreement tracks correctness. That coupling is rarely measured directly: natural-image perturbations preserve meaning only by assumption, and no exact answer key localizes errors. Scientific figures remove both obstacles, a figure is drawn from data by a program, so redrawing it yields images that are semantically equivalent by construction and share a programmatically exact answer. We build RENDEQ, a generator of such render-equivalence sets, and measure the coupling on three open-weight VLMs, checking every finding across three independent instantiations. Re-rendering beats resampling on both accuracy and reliability. Agreement beats an evidence-carrying baseline, mean token log-probability, on two of three models and ties on the third, reversing an intermediate, buggy replication traced to a rendering-pipeline failure. The dispersion behind this is concentrated in one style factor, the plotting library, more than double the next-largest factor and an order of magnitude above the noise floor. Fine-tuning on the model's own cross-render consensus inverts: accuracy falls in every one of five replication runs, the opposite sign to published results on natural images. Agreement certifies correctness only above a threshold set by how diffuse a model's errors are, and an objective that rewards agreement destroys exactly that diffuseness.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 36 references that are in the map. First posted on arXiv 6 Aug 2026.
Builds on
29 of 36 references.
References · 29
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
- 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
- 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
- 2019PlotQA: Reasoning over Scientific Plotscited by 321
- 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
- 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 73
- 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 67
- 2025Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Rewardcited by 45
- 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoningcited by 40
17 more
- 2024Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answeringcited by 32
- 2024Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustnesscited by 17
- 2025Efficient Test-Time Scaling for Small Vision-Language Modelscited by 12
- 2026On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMscited by 11
- 2026ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratchcited by 10
- 2025ChartGen: Scaling Chart Understanding Via Code-Guided Synthetic Chart Generationcited by 8
- 2026VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoningcited by 8
- 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmarkcited by 8
- 2024Decompose and Compare Consistency: Measuring VLMs' Answer Reliability via Task-Decomposition Consistency Comparisoncited by 7
- 2025Chart-RVR: Reinforcement Learning with Verifiable Rewards for Explainable Chart Reasoningcited by 7
- 2026V-FAT: Benchmarking Visual Fidelity Against Text-biascited by 6
- 2025Losing the Plot: How VLM responses degrade on imperfect chartscited by 5
- 2025Questioning the Stability of Visual Question Answeringcited by 5
- 2026VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluationcited by 5
- 2026CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Modelscited by 4
- 2026Zoom Consistency: A Free Confidence Signal in Multi-Step Visual Grounding Pipelinescited by 4
- 2026Chart-RL: Generalized Chart Comprehension via Reinforcement Learning with Verifiable Rewardscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification
- 2026CARGO-VL: Counterfactual Arbitration with Risk-Constrained Group Optimization for Vision-Language Models
- 2026Robust for the Wrong Reasons: The Representational Geometry of LLM Robustness to Science Skepticism
- 2026HalluPrism: When Multimodal Uncertainty Should Diagnose, Not Decide
- 2026Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading
- 2026When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals
- 2026Wrong Before Right: Late Rescue and Interface Failure in Aligned Language Models
- 2026Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation