# Advancing Sequential Numerical Prediction in Autoregressive Models

arXiv: https://arxiv.org/abs/2505.13077 · HTML: https://mlsanity.com/paper/2505.13077/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 54 references that are in the map.
- First posted on arXiv 19 May 2025.

## Builds on (most cited first, 30 of 37)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://mlsanity.com/paper/2402.14804/) (2024)
- [Optimal Transport for Domain Adaptation](https://mlsanity.com/paper/1507.00504/) (2015)
- [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019)
- [Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognition](https://mlsanity.com/paper/1710.10400/) (2017)
- [FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions](https://mlsanity.com/paper/2004.05565/) (2020)
- [MAGE: Machine-generated Text Detection in the Wild](https://mlsanity.com/paper/2305.13242/) (2023)
- [Detecting Curve Text in the Wild: New Dataset and New Solution](https://mlsanity.com/paper/1712.02170/) (2017)
- [A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning](https://mlsanity.com/paper/2110.01515/) (2021)
- [Aligning Large Language Models with Human Preferences through Representation Engineering](https://mlsanity.com/paper/2312.15997/) (2023)
- [A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding](https://mlsanity.com/paper/2407.01976/) (2024)
- [Squared Earth Mover's Distance-based Loss for Training Deep Neural Networks](https://mlsanity.com/paper/1611.05916/) (2016)
- [DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents](https://mlsanity.com/paper/2311.09805/) (2023)
- [SCORE: Story Coherence and Retrieval Enhancement for AI Narratives](https://mlsanity.com/paper/2503.23512/) (2025)

## Built on by (most cited first, 26 of 26)

- [Prolonged Reasoning Is Not All You Need: Certainty-Based Adaptive Routing for Efficient LLM/MLLM Reasoning](https://mlsanity.com/paper/2505.15154/) (2025)
- [SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving](https://mlsanity.com/paper/2512.10719/) (2025)
- [Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by Refining Belief States](https://mlsanity.com/paper/2510.11052/) (2025)
- [White-Balance First, Adjust Later: Cross-Camera Color Constancy via Vision-Language Evaluation](https://mlsanity.com/paper/2605.19613/) (2026)
- [Enhancing Numerical Prediction in LLMs via Smooth MMD Alignment](https://mlsanity.com/paper/2606.27731/) (2026)
- [LPCAN: Lightweight Pyramid Cross-Attention Network for Rail Surface Defect Detection Using RGB-D Data](https://mlsanity.com/paper/2601.09118/) (2026)
- [Knowledge-Embedded and Hypernetwork-Guided Few-Shot Substation Meter Defect Image Generation Method](https://mlsanity.com/paper/2601.09238/) (2026)
- [MA-LipNet: Multi-Dimensional Attention Networks for Robust Lipreading](https://mlsanity.com/paper/2601.20881/) (2026)
- [The Diffusion Duet: Harmonizing Dual Channels with Wavelet Suppression for Image Separation](https://mlsanity.com/paper/2602.13361/) (2026)
- [Detecting Deepfakes with Multivariate Soft Blending and CLIP-based Image-Text Alignment](https://mlsanity.com/paper/2602.15903/) (2026)
- [Efficient Image Super-Resolution with Multi-Scale Spatial Adaptive Attention Networks](https://mlsanity.com/paper/2603.00118/) (2026)
- [A Deformable Attention-Based Detection Transformer with Cross-Scale Feature Fusion for Industrial Coil Spring Inspection](https://mlsanity.com/paper/2603.13429/) (2026)
- [Multi-View Deformable Convolution Meets Visual Mamba for Coronary Artery Segmentation](https://mlsanity.com/paper/2603.21829/) (2026)
- [JND-Guided Neural Watermarking with Spatial Transformer Decoding for Screen-Capture Robustness](https://mlsanity.com/paper/2603.26766/) (2026)
- [Transmittance-Guided Structure-Texture Decomposition for Nighttime Image Dehazing](https://mlsanity.com/paper/2603.29507/) (2026)
- [Hierarchical Awareness Adapters with Hybrid Pyramid Feature Fusion for Dense Depth Prediction](https://mlsanity.com/paper/2604.03339/) (2026)
- [Lightweight Real-Time Rendering Parameter Optimization via XGBoost-Driven Lookup Tables](https://mlsanity.com/paper/2604.25178/) (2026)
- [Image Classification via Random Dilated Convolution with Multi-Branch Feature Extraction and Context Excitation](https://mlsanity.com/paper/2604.25188/) (2026)
- [Multi-Branch Non-Homogeneous Image Dehazing via Concentration Partitioning and Image Fusion](https://mlsanity.com/paper/2605.00885/) (2026)
- [Local Spatiotemporal Convolutional Network for Robust Gait Recognition](https://mlsanity.com/paper/2605.14548/) (2026)
- [Do You Need Text Rectification? Soft Attention Mask Embedding for Rectification-Free Scene Text Spotting](https://mlsanity.com/paper/2605.18173/) (2026)
- [DAIN: Dynamic Agent-Based Interaction Network for Efficient and Collaborative Multimodal Reasoning](https://mlsanity.com/paper/2606.30189/) (2026)
- [ProWAFT: A ROMA-LPD Instance for Workload-Aware and Dynamic Fault Tolerance in FPGA-Based CNN Accelerators](https://mlsanity.com/paper/2607.01602/) (2026)
- [MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models](https://mlsanity.com/paper/2608.03740/) (2026)
- [When Tokenization is Secretly Output Supervision](https://mlsanity.com/paper/2609.01386/) (2026)
- [Contour-Guided Spectral Routing for Robust Real-Time Pedestrian Detection](https://mlsanity.com/paper/2609.14383/) (2026)

## Abstract (arXiv)

Autoregressive models have become the de facto choice for sequence generation tasks, but standard approaches treat digits as independent tokens and apply cross-entropy loss, overlooking the coherent structure of numerical sequences. This paper introduces Numerical Token Integrity Loss (NTIL) to address this gap. NTIL operates at two levels: (1) token-level, where it extends the Earth Mover's Distance (EMD) to preserve ordinal relationships between numerical values, and (2) sequence-level, where it penalizes the overall discrepancy between the predicted and actual sequences. This dual approach improves numerical prediction and integrates effectively with LLMs/MLLMs. Extensive experiments show significant performance improvements with NTIL.