mlsanityData to 14 Oct 2026

Sparse Token Routing in Efficient Transformers

arXiv 2608.20632 ↗First posted 21 Aug 2026cs.CLcited by 0

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Efficient-transformer research often motivates token pruning and adaptive computation with the claim that not all tokens require equal computational effort. We test this claim end to end using SEWN, a two-stream Transformer that routes tokens through either lightweight or full-capacity processing using a learned gate. Across our experiments, routing introduces negligible accuracy change relative to parameter-matched baselines, while the gate's token-importance signal depends critically on how it is learned. A static lexicon-seeded prior fails a counterfactual faithfulness test on BoolQ, whereas a fully contextual gate achieves highly significant separation (p<10-10) on both evaluated tasks without changing task accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 23 references that are in the map. First posted on arXiv 21 Aug 2026.

Builds on

17 of 23 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Token Merging: Your ViT But Faster17 Oct
  2. 2021Learned Token Pruning for Transformers2 Jul

Other references · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  5. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  6. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  7. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  8. 2019Attention is not Explanationcited by 929
  9. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  10. 2020Quantifying Attention Flow in Transformerscited by 775
  11. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
  12. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
3 more
  1. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  2. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  3. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inferencecited by 40

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Adaptive Computation Depth via Learned Token Routing in Transformers
  2. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Models
  3. 2025Pause Tokens Strictly Increase the Expressivity of Constant-Depth Transformers
  4. 2026Mixture of Chapters: Scaling Learnt Memory in Transformers
  5. 2026The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
  6. 2025SliceMoE: Routing Embedding Slices Instead of Tokens for Fine-Grained and Balanced Transformer Scaling
  7. 2026Directional Routing in Transformers
  8. 2026Grounding latent algorithm routing in transformer reasoning