JET: Justification Evaluation in Transformer
Not on the map. Why?
Abstract (arXiv)
JET uses pretrained language and vision-language models to select among a finite set of answers without additional training. It evaluates candidate likelihoods directly and shares computation across candidates. Experiments on desktop CPUs and consumer GPUs assess decision accuracy and execution cost. Qwen3.6-35B-A3B achieves 87.48% accuracy on the full MMLU test set and 3.69 requests per second on a separately timed MMLU subset. The accuracy-throughput comparison covers model, hardware, and reasoning choices, with Jev as an external reference. Controlled execution experiments show 2.18-2.23-fold speedups from prefix reuse and cache management, and a 30.8% reduction in process time from input preparation optimizations, with unchanged outputs. Optional reasoning has a task-dependent accuracy-throughput trade-off. These results support local decision inference from existing models.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 8 of its 15 references that are in the map. First posted on arXiv 27 Sep 2026.
Builds on
8 of 15 references.
References · 8
- 2017Attention Is All You Needcited by 60,361
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
- 2015A large annotated corpus for learning natural language inferencecited by 1,847
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inference
- 2026Dropout Robustness and Cognitive Profiling of Transformer Models via Stochastic Inference
- 2024MELTing point: Mobile Evaluation of Language Transformers
- 2024Do Efficient Transformers Really Save Computation?
- 2026Data Kernel Perspective Space Performance Guarantees for Synthetic Data from Transformer Models
- 2026Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses
- 2025Compute-Accuracy Pareto Frontiers for Open-Source Reasoning Large Language Models
- 2025M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models