mlsanityData to 14 Oct 2026

Trillion 7B Technical Report

arXiv 2504.15431 ↗First posted 21 Apr 2025cs.CL cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

We introduce Trillion-7B, the most token-efficient Korean-centric multilingual LLM available. Our novel Cross-lingual Document Attention (XLDA) mechanism enables highly efficient and effective knowledge transfer from English to target languages like Korean and Japanese. Combined with optimized data mixtures, language-specific filtering, and tailored tokenizer construction, Trillion-7B achieves competitive performance while dedicating only 10% of its 2T training tokens to multilingual data and requiring just 59.4K H100 GPU hours (148K) for full training. Comprehensive evaluations across 27 benchmarks in four languages demonstrate Trillion-7B's robust multilingual performance and exceptional cross-lingual consistency.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 54 of its 59 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

54 of 59 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
38 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  8. 2020GLU Variants Improve Transformercited by 1,804
  9. 2019Root Mean Square Layer Normalizationcited by 1,480
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  11. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  12. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  13. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  14. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  15. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  16. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  17. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  18. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  19. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
  20. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  21. 2024How do Large Language Models Handle Multilingualism?cited by 150
  22. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  23. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  24. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  25. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  26. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  27. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 76
  28. 2024Does your data spark joy? Performance gains from domain upsampling at the end of trainingcited by 52
  29. 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Modelscited by 47
  30. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gapcited by 43
  31. 2022KOBEST: Korean Balanced Evaluation of Significant Taskscited by 37
  32. 2024The broader spectrum of in-context learningcited by 35
  33. 2021The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Designcited by 33
  34. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28
  35. 2023Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Abilitycited by 27
  36. 2024Code-Switching Curriculum Learning for Multilingual Transfer in LLMscited by 27
  37. 2024Analysing The Impact of Sequence Composition on Language Model Pre-Trainingcited by 25
  38. 2023A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Modelscited by 22

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Predicting LLM Reasoning Performance with Small Proxy Modelcited by 5
  2. 2025KORMo: Korean Open Reasoning Model for Everyonecited by 4
  3. 2025Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networkscited by 2
  4. 2026Do LLMs Need Inherent Reasoning Before Reinforcement Learning? A Study in Korean Self-Correctioncited by 1