mlsanityData to 7 Oct 2026

TeleChat Technical Report

arXiv 2401.03804 ↗First posted 8 Jan 2024cs.CL cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

In this technical report, we present TeleChat, a collection of large language models (LLMs) with parameters of 3 billion, 7 billion and 12 billion. It includes pretrained language models as well as fine-tuned chat models that is aligned with human preferences. TeleChat is initially pretrained on an extensive corpus containing a diverse collection of texts from both English and Chinese languages, including trillions of tokens. Subsequently, the model undergoes fine-tuning to align with human preferences, following a detailed methodology that we describe. We evaluate the performance of TeleChat on various tasks, including language understanding, mathematics, reasoning, code generation, and knowledge-based question answering. Our findings indicate that TeleChat achieves comparable performance to other open-source models of similar size across a wide range of public benchmarks. To support future research and applications utilizing LLMs, we release the fine-tuned model checkpoints of TeleChat's 7B and 12B variant, along with code and a portion of our pretraining data, to the public community.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 54 of its 65 references that are in the map. First posted on arXiv 8 Jan 2024.

Builds on

54 of 65 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2017Proximal Policy Optimization Algorithmscited by 14,441
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  11. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  12. 2021Evaluating Large Language Models Trained on Codecited by 8,053
38 more
  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  13. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  14. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  15. 2020GLU Variants Improve Transformercited by 1,724
  16. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  17. 2019Root Mean Square Layer Normalizationcited by 1,425
  18. 2017Mixed Precision Trainingcited by 1,135
  19. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  20. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  21. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  22. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  23. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  24. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  25. 2016Language Modeling with Gated Convolutional Networkscited by 688
  26. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  27. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  28. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  29. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  30. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  31. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  32. 2023Tool Learning with Foundation Modelscited by 302
  33. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  34. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 227
  35. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  36. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  37. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  38. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 5.

15 papers.

Sort:

By area:

  1. 2025Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Datacited by 35
  2. 2024Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Modelcited by 25
  3. 2024CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Modelscited by 11
  4. 2025MR-UIE: Multi-Perspective Reasoning with Reinforcement Learning for Universal Information Extractioncited by 6
  5. 2025Emotional Support with LLM-based Empathetic Dialogue Generationcited by 4
  6. 2025TableZoomer: A Collaborative Agent Framework for Large-scale Table Question Answeringcited by 3
  7. 2025TableReasoner: Advancing Table Reasoning Framework with Large Language Modelscited by 2
  8. 2025GOAT-SLM: A Spoken Language Model with Paralinguistic and Speaker Characteristic Awarenesscited by 2
  9. 2025TeleEval-OS: Performance evaluations of large language models for operations schedulingcited by 1
  10. 2026ReasonTabQA: A Comprehensive Benchmark for Table Question Answering from Real World Industrial Scenarioscited by 1
  11. 2025Multi-Intent Spoken Language Understanding: Methods, Trends, and Challengescited by 0
  12. 2026A benchmark for joint dialogue satisfaction, emotion recognition, and emotion state transition predictioncited by 0
3 more
  1. 2026Dynamic Knowledge Fusion for Multi-Domain Dialogue State Trackingcited by 0
  2. 2026Enhancing Confidence Estimation in Telco LLMs via Twin-Pass CoT-Ensemblingcited by 0
  3. 2026Multi-Objective Exploration and Preference Optimization via Mutual Informationcited by 0