mlsanityData to 7 Oct 2026

52B to 1T: Lessons Learned via Tele-FLM Series

arXiv 2407.02783 ↗First posted 3 Jul 2024cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) represent a significant stride toward Artificial General Intelligence. As scaling laws underscore the potential of increasing model sizes, the academic community has intensified its investigations into LLMs with capacities exceeding 50 billion parameters. This technical report builds on our prior work with Tele-FLM (also known as FLM-2), a publicly available 52-billion-parameter model. We delve into two primary areas: we first discuss our observation of Supervised Fine-tuning (SFT) on Tele-FLM-52B, which supports the "less is more" approach for SFT data construction; second, we demonstrate our experiments and analyses on the best practices for progressively growing a model from 52 billion to 102 billion, and subsequently to 1 trillion parameters. We will open-source a 1T model checkpoint, namely Tele-FLM-1T, to advance further training and research.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 21 of its 27 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

21 of 27 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2023Mistral 7Bcited by 3,065
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2020GLU Variants Improve Transformercited by 1,724
  11. 2019Root Mean Square Layer Normalizationcited by 1,425
  12. 2024Yi: Open Foundation Models by 01.AIcited by 706
9 more
  1. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  2. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  3. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  4. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  5. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  6. 2023Masked Structural Growth for 2x Faster Language Model Pre-trainingcited by 32
  7. 2023ResiDual: Transformer with Dual Residual Connectionscited by 30
  8. 2023FLM-101B: An Open LLM and How to Train It with 100K Budgetcited by 22
  9. 2023nanoLM: an Affordable LLM Pre-training Benchmark via Accurate Loss Prediction across Scalescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 4.

11 papers.

  1. 2025Vision-centric Token Compression in Large Language Modelcited by 21
  2. 2025Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Modelscited by 10
  3. 2025MR-UIE: Multi-Perspective Reasoning with Reinforcement Learning for Universal Information Extractioncited by 6
  4. 2025Training Report of TeleChat3-MoEcited by 4
  5. 2024Open-domain Implicit Format Control for Large Language Model Generationcited by 2
  6. 2025Introducing Visual Scenes and Reasoning: A More Realistic Benchmark for Spoken Language Understandingcited by 1
  7. 2026Hallucinations as Orthogonal Noise: Inference-Time Manifold Alignment via Dynamic Contextual Orthogonalizationcited by 1
  8. 2025Multi-Intent Spoken Language Understanding: Methods, Trends, and Challengescited by 0
  9. 2026Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinationscited by 0
  10. 2026Resonant Context Anchoring: Decoupling Attention Routing and Signal Gain at Inference Timecited by 0
  11. 2026Multi-Objective Exploration and Preference Optimization via Mutual Informationcited by 0