mlsanityData to 7 Oct 2026

YUAN 2.0: A Large Language Model with Localized Filtering-based Attention

arXiv 2311.15786 ↗First posted 27 Nov 2023cs.CL cs.AI cs.HCcited by 9

Not on the map. Why?

Abstract (arXiv)

In this work, we develop and release Yuan 2.0, a series of large language models with parameters ranging from 2.1 billion to 102.6 billion. The Localized Filtering-based Attention (LFA) is introduced to incorporate prior knowledge of local dependencies of natural language into Attention. A data filtering and generating system is presented to build pre-training and fine-tuning dataset in high quality. A distributed training method with non-uniform pipeline parallel, data parallel, and optimizer parallel is proposed, which greatly reduces the bandwidth requirements of intra-node communication, and achieves good performance in large-scale distributed training. Yuan 2.0 models display impressive ability in code generation, math problem-solving, and chatting compared with existing models. The latest version of YUAN 2.0, including model weights and source code, is accessible at Github.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

42 of 56 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
30 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  12. 2023BloombergGPT: A Large Language Model for Financecited by 923
  13. 2023StarCoder: may the source be with you!cited by 790
  14. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  15. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  16. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  17. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  18. 2023Textbooks Are All You Needcited by 561
  19. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  20. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  21. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  22. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  23. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  24. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 151
  25. 2020Ape210K: A Large-Scale and Template-Rich Dataset of Math Word Problemscited by 60
  26. 2023WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Modelscited by 48
  27. 2021Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learningcited by 35
  28. 2023Automatic Model Selection with Large Language Models for Reasoningcited by 35
  29. 2023MFTCoder: Boosting Code LLMs with Multitask Fine-Tuningcited by 34
  30. 2023Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 0; 2026: 3.

9 papers.

  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  2. 2024Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactionscited by 43
  3. 2024SimpleLLM4AD: An End-to-End Vision-Language Model with Graph Visual Question Answering for Autonomous Drivingcited by 16
  4. 2024HiFT: A Hierarchical Full Parameter Fine-Tuning Strategycited by 12
  5. 2024Revolutionizing Database Q&A with Large Language Models: Comprehensive Benchmark and Evaluationcited by 7
  6. 2026FBS: Modeling Native Parallel Reading inside a Transformercited by 6
  7. 2024HeGraphAdapter: Tuning Multi-Modal Vision-Language Models with Heterogeneous Graph Adaptercited by 3
  8. 2026Yuan3.0 Flash: An Open Multimodal Large Language Model for Enterprise Applicationscited by 2
  9. 2026Yuan3.0 Ultra: A Trillion-Parameter Enterprise-Oriented MoE LLMcited by 0