mlsanityData to 7 Oct 2026

AnyTrans: Translate AnyText in the Image with Large Scale Models

arXiv 2406.11432 ↗First posted 17 Jun 2024cs.CV cs.AIcited by 8

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

This paper introduces AnyTrans, an all-encompassing framework for the task-Translate AnyText in the Image (TATI), which includes multilingual text translation and text fusion within images. Our framework leverages the strengths of large-scale models, such as Large Language Models (LLMs) and text-guided diffusion models, to incorporate contextual cues from both textual and visual elements during translation. The few-shot learning capability of LLMs allows for the translation of fragmented texts by considering the overall context. Meanwhile, the advanced inpainting and editing abilities of diffusion models make it possible to fuse translated text seamlessly into the original image while preserving its style and realism. Additionally, our framework can be constructed entirely using open-source models and requires no training, making it highly accessible and easily expandable. To encourage advancement in the TATI task, we have meticulously compiled a test dataset called MTIT6, which consists of multilingual text image translation data from six language pairs.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 50 of its 68 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

50 of 68 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering28 Nov
  2. 2023TextDiffuser: Diffusion Models as Text Painters18 May
  3. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  5. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Models23 Nov
  6. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  8. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  9. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 41

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2014Conditional Generative Adversarial Netscited by 2,152
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  7. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  8. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  9. 2021Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contractioncited by 306
  10. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 278
  11. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  12. 2017EAST: An Efficient and Accurate Scene Text Detectorcited by 248
29 more
  1. 2019Real-time Scene Text Detection with Differentiable Binarizationcited by 212
  2. 2017Arbitrary-Oriented Scene Text Detection via Rotation Proposalscited by 176
  3. 2022Scene Text Recognition with Permuted Autoregressive Sequence Modelscited by 148
  4. 2020PP-OCR: A Practical Ultra Lightweight OCR Systemcited by 147
  5. 2019ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019cited by 138
  6. 2016Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanismcited by 117
  7. 2017Multi-Content GAN for Few-Shot Font Style Transfercited by 100
  8. 2018Rapid Adaptation of Neural Machine Translation to New Languagescited by 82
  9. 2019Editing Text in the Wildcited by 81
  10. 2023Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Modelscited by 61
  11. 2021Benchmarking Chinese Text Recognition: Datasets, Baselines, and an Empirical Studycited by 52
  12. 2023TIM: Teaching Large Language Models to Translate with Comparisoncited by 50
  13. 2017Doubly-Attentive Decoder for Multi-modal Neural Machine Translationcited by 46
  14. 2018Multi-Oriented Scene Text Detection via Corner Localization and Region Segmentationcited by 42
  15. 2017Imagination improves Multimodal Translationcited by 38
  16. 2021MOST: A Multi-Oriented Scene Text Detector with Localization Refinementcited by 33
  17. 2017Attention Strategies for Multi-Source Sequence-to-Sequence Learningcited by 32
  18. 2023GraphControl: Adding Conditional Control to Universal Graph Pre-trained Models for Graph Domain Transfer Learningcited by 32
  19. 2019Multimodal Machine Translation through Visuals and Speechcited by 29
  20. 2021Language Tags Matter for Zero-Shot Neural Machine Translationcited by 29
  21. 2023Chinese Text Recognition with A Pre-Trained CLIP-Like Model Through Image-IDS Aligningcited by 24
  22. 2017A Teacher-Student Framework for Zero-Resource Neural Machine Translationcited by 22
  23. 2018Input Combination Strategies for Multi-Source Transformer Decodercited by 22
  24. 2020Towards End-to-End In-Image Neural Machine Translationcited by 20
  25. 2022Improving End-to-End Text Image Translation From the Auxiliary Text Translation Taskcited by 20
  26. 2016Multimodal Attention for Neural Machine Translationcited by 19
  27. 2018TET-GAN: Text Effects Transfer via Stylization and Destylizationcited by 19
  28. 2018Zero-Resource Neural Machine Translation with Multi-Agent Communication Gamecited by 8
  29. 2021StrokeNet: Stroke Assisted and Hierarchical Graph Reasoning Networkscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2026InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editingcited by 23
  2. 2025ComfyGPT: A Self-Optimizing Multi-Agent System for Comprehensive ComfyUI Workflow Generationcited by 14
  3. 2025Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptationcited by 3
  4. 2026Global-Local Dual Perception for MLLMs in High-Resolution Text-Rich Image Translationcited by 2
  5. 2026ICDAR 2025 Competition on End-to-End Document Image Machine Translation Towards Complex Layoutscited by 1
  6. 2026VaaWIT: Visual-Aware Adaptation of Large Language Models for Multilingual Web Image Translationcited by 0
  7. 2026Comparative Evaluation of Machine Translation Systems on Images with Textcited by 0
  8. 2026VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubricscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023AnyText: Multilingual Visual Text Generation And Editing
  2. 2025TextFlux: An OCR-Free DiT Model for High-Fidelity Multilingual Scene Text Synthesis
  3. 2024Ensuring Consistency for In-Image Translation
  4. 2023AltDiffusion: A Multilingual Text-to-Image Diffusion Model
  5. 2026MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation
  6. 2025LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven Language Representation
  7. 2024MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
  8. 2023Conditional Score Guidance for Text-Driven Image-to-Image Translation