NeoBabel: A Multilingual Open Tower for Visual Generation
Not on the map. Why?
Abstract (arXiv)
Text-to-image generation advancements have been predominantly English-centric, creating barriers for non-English speakers and perpetuating digital inequities. While existing systems rely on translation pipelines, these introduce semantic drift, computational overhead, and cultural misalignment. We introduce NeoBabel, a novel multilingual image generation framework that sets a new Pareto frontier in performance, efficiency and inclusivity, supporting six languages: English, Chinese, Dutch, French, Hindi, and Persian. The model is trained using a combination of large-scale multilingual pretraining and high-resolution instruction tuning. To evaluate its capabilities, we expand two English-only benchmarks to multilingual equivalents: m-GenEval and m-DPG. NeoBabel achieves state-of-the-art multilingual performance while retaining strong English capability, scoring 0.75 on m-GenEval and 0.68 on m-DPG. Notably, it performs on par with leading models on English tasks while outperforming them by +0.11 and +0.09 on multilingual benchmarks, even though these models are built on multilingual base LLMs. This demonstrates the effectiveness of our targeted alignment training for preserving and extending crosslingual generalization. We further introduce two new metrics to rigorously assess multilingual alignment and robustness to code-mixed prompts. Notably, NeoBabel matches or exceeds English-only models while being 2-4x smaller. We release an open toolkit, including all code, model checkpoints, a curated dataset of 124M multilingual text-image pairs, and standardized multilingual evaluation protocols, to advance inclusive AI research. Our work demonstrates that multilingual capability is not a trade-off but a catalyst for improved robustness, efficiency, and cultural fidelity in generative AI.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 85 of its 88 references that are in the map. First posted on arXiv 8 Jul 2025.
Builds on
85 of 88 references.
References · 85
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Segment Anythingcited by 8,234
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2023Visual Instruction Tuningcited by 7,869
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
- 2022Scalable Diffusion Models with Transformerscited by 6,054
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2022Classifier-Free Diffusion Guidancecited by 5,847
- 2022Flow Matching for Generative Modelingcited by 5,702
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2024Qwen2.5 Technical Reportcited by 4,942
38 more
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
- 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
- 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
- 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
- 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
- 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
- 2024Emu3: Next-Token Prediction is All You Needcited by 781
- 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
- 2024Autoregressive Image Generation without Vector Quantizationcited by 651
- 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
- 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
- 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 502
- 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
- 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
- 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
- 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
- 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 376
- 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
- 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 302
- 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
- 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
- 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
- 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 282
- 2020Query-Key Normalization for Transformerscited by 280
- 2025Transfer between Modalities with MetaQueriescited by 236
- 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
- 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 205
- 2023Any-to-Any Generation via Composable Diffusioncited by 204
50 most cited of 85 All 85 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.