mlsanityData to 7 Oct 2026

A Sovereign, Open-Source Foundation Model for German and English

arXiv 2607.09424 ↗First posted 10 Jul 2026cs.CL cs.AI cs.LGcited by 1

On the map: Mixture-of-experts → MoE large language models

also in 1 more
Abstract (arXiv)

We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English. Its hybrid design activates only 3B of 30B parameters per token and keeps the inference cache near-constant as context grows, giving it a decisive throughput advantage over dense models for long-context, high-concurrency deployment. Pretrained on roughly 27 trillion tokens with deliberately up-weighted German, Soofi S matches dense 14 to 27B models on aggregate English and German benchmarks while achieving the best code aggregates in both languages among 17 open base models, and outperforms every European sovereign baseline in our comparison, including ones far larger in active parameters. Among fully open models, Soofi S obtains the highest English and German evaluation scores, ahead of Olmo 3 32B and Apertus 70B. Soofi S was built end-to-end on the German Industrial AI Cloud, a sovereign HPC scale AI infrastructure operated by Deutsche Telekom in Munich. Soofi S will be released under highly permissive, open-access terms: weights, selected intermediate checkpoints, full per-source data accounting, hyperparameters, and training and evaluation code. Where source licenses permit, data-construction artifacts are released under permissive licenses; commercially licensed sources are documented with aggregate statistics and exact mixture accounting.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. First posted on arXiv 10 Jul 2026.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2024Jamba: A Hybrid Transformer-Mamba Language Model
  2. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
  3. 2024An Empirical Study of Mamba-based Language Models
  4. 2024Jamba-1.5: Hybrid Transformer-Mamba Models at Scale
  5. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models
  6. 2024BlackMamba: Mixture of Experts for State-Space Models
  7. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  8. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Models