mlsanityData to 7 Oct 2026

Tele-FLM Technical Report

arXiv 2404.16645 ↗First posted 25 Apr 2024cs.CL cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have showcased profound capabilities in language understanding and generation, facilitating a wide array of applications. However, there is a notable paucity of detailed, open-sourced methodologies on efficiently scaling LLMs beyond 50 billion parameters with minimum trial-and-error cost and computational resources. In this report, we introduce Tele-FLM (aka FLM-2), a 52B open-sourced multilingual large language model that features a stable, efficient pre-training paradigm and enhanced factual judgment capabilities. Tele-FLM demonstrates superior multilingual language modeling abilities, measured by BPB on textual corpus. Besides, in both English and Chinese foundation model evaluation, it is comparable to strong open-sourced models that involve larger pre-training FLOPs, such as Llama2-70B and DeepSeek-67B. In addition to the model weights, we share the core designs, engineering practices, and training details, which we expect to benefit both the academic and industrial communities.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 61 of its 78 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

61 of 78 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2017Proximal Policy Optimization Algorithmscited by 14,441
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  11. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  12. 2021Evaluating Large Language Models Trained on Codecited by 8,053
38 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2023Mistral 7Bcited by 3,065
  12. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  13. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  14. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  15. 2022Emergent Abilities of Large Language Modelscited by 2,483
  16. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  17. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  18. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  19. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  20. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  21. 2020GLU Variants Improve Transformercited by 1,724
  22. 2019Root Mean Square Layer Normalizationcited by 1,425
  23. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  24. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  25. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  26. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  27. 2024Yi: Open Foundation Models by 01.AIcited by 706
  28. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  29. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  30. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  31. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  32. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  33. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  34. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  35. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  36. 2016FastText.zip: Compressing text classification modelscited by 331
  37. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  38. 2023Language Modeling Is Compressioncited by 225

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 1.

12 papers.

  1. 2025FLM-Audio: Natural Monologues Improves Native Full-Duplex Chatbots via Dual Trainingcited by 9
  2. 2025Scaling Diffusion Transformers Efficiently via μPcited by 6
  3. 2025MR-UIE: Multi-Perspective Reasoning with Reinforcement Learning for Universal Information Extractioncited by 6
  4. 2025Emotional Support with LLM-based Empathetic Dialogue Generationcited by 4
  5. 2025Training Report of TeleChat3-MoEcited by 4
  6. 2024Sketch: A Toolkit for Streamlining LLM Operationscited by 3
  7. 2025TableZoomer: A Collaborative Agent Framework for Large-scale Table Question Answeringcited by 3
  8. 2024Open-domain Implicit Format Control for Large Language Model Generationcited by 2
  9. 2024Athena: Retrieval-augmented Legal Judgment Prediction with Large Language Modelscited by 2
  10. 2025TableReasoner: Advancing Table Reasoning Framework with Large Language Modelscited by 2
  11. 2025Multi-Intent Spoken Language Understanding: Methods, Trends, and Challengescited by 0
  12. 2026Multi-Objective Exploration and Preference Optimization via Mutual Informationcited by 0