mlsanityData to 7 Oct 2026

You Only Need Your Transformer 25% of the Time: Meaning-First Execution for Eliminating Unnecessary Inference

arXiv 2601.00847 ↗First posted 29 Dec 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Modern AI inference systems treat transformer execution as mandatory, conflating model capability with execution necessity. We reframe inference as a control-plane decision problem: determining when execution is necessary versus when correctness can be preserved through alternative pathways. We introduce Meaning-First Execution (MFEE), a control-plane architecture implementing this framework, selectively invoking transformer inference only when required. MFEE operates as a gating layer above existing stacks without modifying models, weights, or parameters. Across 1,000 diverse prompts under deterministic decoding, MFEE achieves 78.1% execution reduction while maintaining 100% exact-match equivalence for invoked executions. Comparative evaluation reveals pattern-based routers achieve at most 53.3% avoidance with correctness failures, while MFEE reaches 100% avoidance with zero failures through semantic analysis. We prove this limitation via Theorem 1: any router operating solely on finite feature maps cannot simultaneously guarantee zero false skips and positive avoidance on feature-collision pairs. These results establish execution governance as a foundational layer in ML systems infrastructure, orthogonal to model-level optimization techniques.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 29 Dec 2025.

Builds on

41 of 55 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
29 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  4. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  7. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  8. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  12. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  13. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  14. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  15. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  16. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  17. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  18. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  19. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  20. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  21. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  22. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  23. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  24. 2022Confident Adaptive Language Modelingcited by 297
  25. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  26. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  27. 2019Depth-Adaptive Transformercited by 227
  28. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  29. 2025Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraintscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Position: Artificial Intelligence Needs Meta Intelligence -- the Case for Metacognitive AIcited by 0