mlsanityData to 14 Oct 2026

Image-to-LaTeX Converter for Mathematical Formulas and Text

arXiv 2408.04015 ↗First posted 7 Aug 2024cs.CL cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

In this project, we train a vision encoder-decoder model to generate LaTeX code from images of mathematical formulas and text. Utilizing a diverse collection of image-to-LaTeX data, we build two models: a base model with a Swin Transformer encoder and a GPT-2 decoder, trained on machine-generated images, and a fine-tuned version enhanced with Low-Rank Adaptation (LoRA) trained on handwritten formulas. We then compare the BLEU performance of our specialized model on a handwritten test set with other similar models, such as Pix2Text, TexTeller, and Sumen. Through this project, we contribute open-source models for converting images to LaTeX and provide from-scratch code for building these models with distributed training and GPU optimizations.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 14 of its 30 references that are in the map. First posted on arXiv 7 Aug 2024.

Builds on

14 of 30 references.

References · 14

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  9. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 520
  10. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 279
  11. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 256
  12. 2021MathBERT: A Pre-Trained Model for Mathematical Formula Understandingcited by 58
2 more
  1. 2020Handwritten Optical Character Recognition (OCR): A Comprehensive Systematic Literature Review (SLR)cited by 56
  2. 2020A Survey of Deep Learning Approaches for OCR and Document Understandingcited by 32

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 17
  2. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 3