# Vectra: A New Metric, Dataset, and Model for Visual Quality Assessment in E-Commerce In-Image Machine Translation

arXiv: https://arxiv.org/abs/2602.07014 · HTML: https://mlsanity.com/paper/2602.07014/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 48 references that are in the map.
- First posted on arXiv 31 Jan 2026.

## Builds on (most cited first, 25 of 25)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning](https://mlsanity.com/paper/2507.01006/) (2025)
- [MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency](https://mlsanity.com/paper/2502.09621/) (2025)
- [Exploring Better Text Image Translation with Multimodal Codebook](https://mlsanity.com/paper/2305.17415/) (2023)
- [Towards End-to-End In-Image Neural Machine Translation](https://mlsanity.com/paper/2010.10648/) (2020)
- [Improving End-to-End Text Image Translation From the Auxiliary Text Translation Task](https://mlsanity.com/paper/2210.03887/) (2022)
- [Similarity and Quality Metrics for MR Image-To-Image Translation](https://mlsanity.com/paper/2405.08431/) (2024)
- [Exploring In-Image Machine Translation with Real-World Background](https://mlsanity.com/paper/2505.15282/) (2025)
- [PRIM: Towards Practical In-Image Multilingual Machine Translation](https://mlsanity.com/paper/2509.05146/) (2025)
- [SAMScore: A Content Structural Similarity Metric for Image Translation Evaluation](https://mlsanity.com/paper/2305.15367/) (2023)
- [Unpaired Image-to-Image Translation with Content Preserving Perspective: A Review](https://mlsanity.com/paper/2502.08667/) (2025)
- [Diffusion Model Compression for Image-to-Image Translation](https://mlsanity.com/paper/2401.17547/) (2024)
- [Multi-Dimensional Machine Translation Evaluation: Model Evaluation and Resource for Korean](https://mlsanity.com/paper/2403.12666/) (2024)
- [Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridge](https://mlsanity.com/paper/2510.20819/) (2025)
- [Ensuring Consistency for In-Image Translation](https://mlsanity.com/paper/2412.18139/) (2024)
- [Ustnlp16 at SemEval-2025 Task 9: Improving Model Performance through Imbalance Handling and Focal Loss](https://mlsanity.com/paper/2505.00021/) (2025)

## Abstract (arXiv)

In-Image Machine Translation (IIMT) powers cross-border e-commerce product listings; existing research focuses on machine translation evaluation, while visual rendering quality is critical for user engagement. When facing context-dense product imagery and multimodal defects, current reference-based methods (e.g., SSIM, FID) lack explainability, while model-as-judge approaches lack domain-grounded, fine-grained reward signals. To bridge this gap, we introduce Vectra, to the best of our knowledge, the first reference-free, MLLM-driven visual quality assessment framework for e-commerce IIMT. Vectra comprises three components: (1) Vectra Score, a multidimensional quality metric system that decomposes visual quality into 14 interpretable dimensions, with spatially-aware Defect Area Ratio (DAR) quantification to reduce annotation ambiguity; (2) Vectra Dataset, constructed from 1.1M real-world product images via diversity-aware sampling, comprising a 2K benchmark for system evaluation, 30K reasoning-based annotations for instruction tuning, and 3.5K expert-labeled preferences for alignment and evaluation; and (3) Vectra Model, a 4B-parameter MLLM that generates both quantitative scores and diagnostic reasoning. Experiments demonstrate that Vectra achieves state-of-the-art correlation with human rankings, and our model outperforms leading MLLMs, including GPT-5 and Gemini-3, in scoring performance. The dataset and model will be released upon acceptance.