mlsanityData to 14 Oct 2026

Token-Operations-Oriented Inference Optimization Techniques for Large Models

arXiv 2606.20295 ↗First posted 18 Jun 2026cs.SE cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large model inference optimization serves as a key foundation for supporting the scalable, low-cost, and highly stable operation of large model services. Centered on token-oriented inference optimization technology, this paper proposes for the first time a four-layer technical architecture consisting of Multi-model Fusion, Model Optimization, Compute-Model Fusion, and Compute-Network-Model Fusion. It systematically reviews the key technologies and current industry status across these four levels and analyzes the application value of related technologies in real-world business scenarios. This paper provides a practical technical path for reducing token production costs, improving token service efficiency, ensuring the stability of token supply, and driving the transition of large model services from being merely callable to being operable.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 123 of its 216 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

123 of 216 references.

References · 123

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
38 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2024Mixtral of Expertscited by 1,823
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  11. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  12. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  13. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  14. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  15. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  16. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  17. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  18. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  19. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  20. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  21. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  22. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  23. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  24. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  25. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  26. 2024DeepSeek-V3 Technical Reportcited by 811
  27. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  28. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  29. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  30. 2022PAL: Program-aided Language Modelscited by 652
  31. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  32. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  33. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  34. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  35. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  36. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  37. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  38. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464

50 most cited of 123 All 123 (JSON)

Built on by

Not cited yet.