mlsanityData to 7 Oct 2026

Yi: Open Foundation Models by 01.AI

arXiv 2403.04652 ↗First posted 7 Mar 2024cs.CL cs.AIcited by 706

Not on the map. Why?

Cited by 147 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce the Yi model family, a series of language and multimodal models that demonstrate strong multi-dimensional capabilities. The Yi model family is based on 6B and 34B pretrained language models, then we extend them to chat models, 200K long context models, depth-upscaled models, and vision-language models. Our base models achieve strong performance on a wide range of benchmarks like MMLU, and our finetuned chat models deliver strong human preference rate on major evaluation platforms like AlpacaEval and Chatbot Arena. Building upon our scalable super-computing infrastructure and the classical transformer architecture, we attribute the performance of Yi models primarily to its data quality resulting from our data-engineering efforts. For pretraining, we construct 3.1 trillion tokens of English and Chinese corpora using a cascaded data deduplication and quality filtering pipeline. For finetuning, we polish a small scale (less than 10K) instruction dataset over multiple iterations such that every single instance has been verified directly by our machine learning engineers. For vision-language, we combine the chat language model with a vision transformer encoder and train the model to align visual representations to the semantic space of the language model. We further extend the context length to 200K through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. We show that extending the depth of the pretrained checkpoint through continual pretraining further improves performance. We believe that given our current results, continuing to scale up model parameters using thoroughly optimized data will lead to even stronger frontier models.

Key facts. Cited by 706 papers in the map (data through 7 Oct 2026). Builds on 69 of its 92 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

69 of 92 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
38 more
  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  13. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  14. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  15. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  16. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  17. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  18. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  19. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  20. 2020GLU Variants Improve Transformercited by 1,724
  21. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  22. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  23. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  24. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  25. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  26. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  27. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  28. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  29. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  30. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  31. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  32. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  33. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  34. 2022Efficiently Scaling Transformer Inferencecited by 529
  35. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  36. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  37. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  38. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 388; 2025: 248; 2026: 70.

706 papers.

Sort:

By area:

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  9. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  10. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  12. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
38 more
  1. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  3. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  4. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  5. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  6. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  7. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  8. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  9. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  10. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  11. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  12. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  13. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  14. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  15. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  16. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  17. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  18. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  19. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  20. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  21. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  22. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  23. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  24. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  25. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  26. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 194
  27. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  28. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  29. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 151
  30. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  31. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
  32. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
  33. 2024TableBench: A Comprehensive and Complex Benchmark for Table Question Answeringcited by 127
  34. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  35. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  36. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  37. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  38. 2024Make Your LLM Fully Utilize the Contextcited by 118

50 most cited of 706 All 706 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
  2. 2023YAYI 2: Multilingual Open-Source Large Language Models
  3. 2025Challenges and Applications of Large Language Models: A Comparison of GPT and DeepSeek family of models
  4. 2023Large Language Models as Automated Aligners for benchmarking Vision-Language Models
  5. 2024PALO: A Polyglot Large Multimodal Model for 5B People
  6. 2023Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages
  7. 2023Ziya-Visual: Bilingual Large Vision-Language Model via Multi-Task Instruction Tuning
  8. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models