mlsanityData to 14 Oct 2026

Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs

arXiv 2605.07731 ↗First posted 8 May 2026cs.CL cs.AIcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

This report benchmarks the performance of ENGINEERING Ingegneria Informatica S.p.A.'s EngGPT2MoE-16B-A3B LLM, a 16B parameter Mixture of Experts (MoE) model with 3B active parameters. Performance is investigated across a wide variety of representative benchmarks, and is compared against comparably-sized open-source MoE and dense models. In comparison with popular Italian models, namely FastwebMIIA-7B, Minerva-7B, Velvet-14B, and LLaMAntino-3-ANITA-8B, EngGPT2MoE-16B-A3B performs as well or better on international benchmarks: ARC-Challenge, GSM8K, AIME24, AIME25, MMLU, and HumanEval (HE). It achieves the best performance for the longest context setting (32k) of the RULER benchmark. On the Italian benchmark dataset ITALIC, the model performs as well or better than the other models except for Velvet-14B, which outperforms it. Compared with popular MoE models of comparable size, the new model reports higher values than DeepSeek-MoE-16B-Chat on all considered benchmarks. It has higher values than Moonlight-16B-A3B on HE, MMLU, AIME24, AIME25, GSM8K, and the 32k RULER setting, but lower on BFCL and some ARC and ITALIC settings. Finally it has lower values than GPT-OSS-20B on most benchmarks, including HE, MMLU, AIME24, AIME25, GSM8K, ARC, BFCL, and the RULER 32k. When compared with popular dense models, EngGPT2MoE-16B-A3B reports higher values on AIME24 and AIME25 than Llama-3.1-8B-Instruct, Gemma-3-12b-it, and Ministral-3-8BInstruct-2512-BF16, but lower values on ITALIC, BFCL, and RULER with a 32k context. When performance is aggregated across all benchmark metrics, EngGPT2MoE-16B-A3B shows higher performance than the Italian models under evaluation while achieving lower results than some of the most performant international models, in particular GPT-5 nano and Qwen3-8B. Taken together, our findings find the new model to be a step forward for native Italian Large Language Models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 42 of its 90 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

42 of 90 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun

Other references · 40

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  10. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  11. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  12. 2025Gemma 3 Technical Reportcited by 1,824
28 more
  1. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  2. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  4. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  5. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  6. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  7. 2025Humanity's Last Examcited by 491
  8. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  9. 2025Muon is Scalable for LLM Trainingcited by 428
  10. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  11. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  12. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  13. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 201
  14. 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 135
  15. 2025CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingscited by 115
  16. 2026Ministral 3cited by 114
  17. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  18. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 106
  19. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
  20. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 86
  21. 2024Controllable Text Generation for Large Language Models: A Surveycited by 57
  22. 2025PolyMath: Evaluating Mathematical Reasoning in Multilingual Contextscited by 56
  23. 2025A Survey of Large Language Model Agents for Question Answeringcited by 48
  24. 2024Needle in the Haystack for Memory Based Large Language Modelscited by 34
  25. 2025AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Modelscited by 31
  26. 2025Sliding Window Attention Training for Efficient Large Language Modelscited by 24
  27. 2025Evalita-LLM: Benchmarking Large Language Models on Italiancited by 7
  28. 2026EngGPT2: Sovereign, Efficient and Open Intelligencecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  2. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems
  3. 2024MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems
  4. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
  5. 2026LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts
  6. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  7. 2025dots.llm1 Technical Report
  8. 2024Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training