mlsanityData to 7 Oct 2026

PACT: Pairwise-Anchored Calibrated Tuning for Single-Token Typed Decisions

arXiv 2609.35865 ↗First posted 26 Sep 2026cs.CL cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Single-token typed-decision models answer a schema question by reading the logits of a few one-letter answer codes at a single position: they are fast and return a probability for every allowed answer, but they are trained with plain cross-entropy that ignores most of the structure in their training data. We study such a model whose data is curated as contrastive pairs---two contexts that differ in one edited fact that flips the answer---each carrying a machine-checked certificate that deleting the decisive sentence makes the fact unknown. We propose PACT, which turns this structure into four training terms that need no new annotation: a difference-in-differences margin over each pair that is invariant to any shared logit offset, a permutation-consistency term against answer-code position bias, an evidence-necessity term on certificate-verified ablated contexts, and an ordinal transport cost for rubric fields, plus a three-parameter contextual temperature. On a frozen 324-item holdout with three seeds, PACT matches the published recipe in accuracy (84.6% vs. 85.2%; McNemar p ≥ 0.50 at every seed) while giving the lowest position bias of all runs (answer flips under relabelling 9.8% vs. 13.8%) and the lowest ordinal error on rubric fields (MAE 0.232 vs. 0.311). Against a control with the same optimiser and schedule but cross-entropy only, PACT is significantly more accurate at two of three seeds, halves the seed-to-seed spread and lowers NLL by 26%. Seed-matched ablations and pre-specified falsification tests locate these gains precisely: no single term raises raw accuracy, and the method's value lies in robustness and stability rather than headline accuracy. Code, data splits, trained adapters, and all run records are available at https://github.com/BennyLinntu/PACT-Pairwise-Anchored-Calibrated-Tuning-for-Single-Token-Typed-Decisions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 39 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

19 of 39 references.

References · 19

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  6. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  7. 2017Selective Classification for Deep Neural Networkscited by 799
  8. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  9. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  10. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  11. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  12. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
7 more
  1. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  2. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  3. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  4. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  5. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  6. 2020Learning What Makes a Difference from Counterfactual Examples and Gradient Supervisioncited by 66
  7. 2016Squared Earth Mover's Distance-based Loss for Training Deep Neural Networkscited by 45

Built on by

Not cited yet.