The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval
Not on the map. Why?
Abstract (arXiv)
Jev's documented interface evaluates typed questions independently against shared evidence. We audit request independence in seven open Jev-style checkpoints on 276 constructed three-passage states from 92 TREC retrieval queries. Paired interventions separately change requested questions, displayed evidence, and candidate eligibility. Request expansion induces strict binary flips in 9.1%/8.0% of Nimble's judgments and changes 21.4%/18.9% of selection-or-abstention actions in TREC DL2019/2020 under a threshold of 0.5 fixed before inference. Correct-action rates change by 0.0/+8.2 percentage points, respectively, with 95% query-bootstrap intervals of [-6.0,6.0] and [3.8,12.6]. Field masking traces all 70 request-set flips to the schema fields serialized into each scoring prompt. In Von, fixing a shared separator's position eliminates two eligibility-induced preference flips in selected diagnostic states. Laya-typed's default CUDA path produces five passage swaps without strict binary flips, and both yearly action-change intervals include zero; in an FP32 rerun, its probability movement vanishes at the model's four-decimal reporting resolution.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 23 references that are in the map. First posted on arXiv 10 Oct 2026.
Builds on
11 of 23 references.
References · 11
- 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
- 2019Passage Re-ranking with BERTcited by 645
- 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
- 2020Overview of the TREC 2019 deep learning trackcited by 286
- 2026Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to Itcited by 25
- 2024Measuring the Inconsistency of Large Language Models in Preferential Rankingcited by 19
- 2026Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understandingcited by 11
- 2026Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failurescited by 8
- 2025Contextual Relevance and Adaptive Sampling for LLM-Based Document Rerankingcited by 4
- 2026Do System One Decisions Add Up? A Study of Probabilistic Coherencecited by 3
- 2026When the Right Answer Is Missing: An Arithmetic-Dependent Rejection Bottleneck in Jevcited by 2
Built on by
Not cited yet.