mlsanityData to 14 Oct 2026

Synergy: End-to-end Concept Model

arXiv 2507.12769 ↗First posted 17 Jul 2025cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we present Synergy, a language model that bridges different levels of abstraction in an end-to-end fashion through a learned routing mechanism. Focusing on low-level linguistic abstraction, we trained our model as a byte-level language model. Our model spontaneously learns to tokenize bytes, producing fewer concept tokens than Byte-level Byte Pair Encoder (BBPE) tokenizers while keeping comparable performance. By comparing with Llama3, we observed an advantage of Synergy under the same model scale and training dataset size. Further studies show that the middle part (the higher abstraction part) of our model performs better when positional encodings are removed, suggesting the emergence of position-independent concepts. These findings demonstrate the feasibility of tokenizer-free architectures, paving the way for more robust and flexible pipelines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 24 references that are in the map. First posted on arXiv 17 Jul 2025.

Builds on

21 of 24 references.

References · 21

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2020GLU Variants Improve Transformercited by 1,804
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  8. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  9. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  10. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  11. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  12. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
9 more
  1. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  2. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 170
  3. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 146
  4. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  5. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 102
  6. 2022Efficient Transformers with Dynamic Token Poolingcited by 76
  7. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  8. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  9. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 28

Built on by

Not cited yet.