# Reverse That Number! Decoding Order Matters in Arithmetic Learning

arXiv: https://arxiv.org/abs/2403.05845 · HTML: https://mlsanity.com/paper/2403.05845/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 16 references that are in the map.
- First posted on arXiv 9 Mar 2024.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [How well do Large Language Models perform in Arithmetic tasks?](https://mlsanity.com/paper/2304.02015/) (2023)
- [Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks](https://mlsanity.com/paper/2305.14201/) (2023)
- [Limitations of Language Models in Arithmetic and Symbolic Induction](https://mlsanity.com/paper/2208.05051/) (2022)
- [GPT Can Solve Mathematical Problems Without a Calculator](https://mlsanity.com/paper/2309.03241/) (2023)
- [Recursion of Thought: A Divide-and-Conquer Approach to Multi-Context Reasoning with Language Models](https://mlsanity.com/paper/2306.06891/) (2023)
- [Evaluating Transformer Language Models on Arithmetic Operations Using Number Decomposition](https://mlsanity.com/paper/2304.10977/) (2023)

## Built on by (most cited first, 9 of 9)

- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models](https://mlsanity.com/paper/2502.11075/) (2025)
- [Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Models](https://mlsanity.com/paper/2411.02083/) (2024)
- [Language Models are Symbolic Learners in Arithmetic](https://mlsanity.com/paper/2410.15580/) (2024)
- [DiffusionGemma Technical Report](https://mlsanity.com/paper/2608.00146/) (2026)
- [Reinforced Context Order Recovery for Adaptive Reasoning and Planning](https://mlsanity.com/paper/2508.13070/) (2025)
- [Language Models Do Not Embed Numbers Continuously](https://mlsanity.com/paper/2510.08009/) (2025)
- [What Makes the Preferred Thinking Direction for LLMs in Multiple-choice Questions?](https://mlsanity.com/paper/2502.18435/) (2025)
- [BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers](https://mlsanity.com/paper/2606.02241/) (2026)

## Abstract (arXiv)

Recent advancements in pretraining have demonstrated that modern Large Language Models (LLMs) possess the capability to effectively learn arithmetic operations. However, despite acknowledging the significance of digit order in arithmetic computation, current methodologies predominantly rely on sequential, step-by-step approaches for teaching LLMs arithmetic, resulting in a conclusion where obtaining better performance involves fine-grained step-by-step. Diverging from this conventional path, our work introduces a novel strategy that not only reevaluates the digit order by prioritizing output from the least significant digit but also incorporates a step-by-step methodology to substantially reduce complexity. We have developed and applied this method in a comprehensive set of experiments. Compared to the previous state-of-the-art (SOTA) method, our findings reveal an overall improvement of in accuracy while requiring only a third of the tokens typically used during training. For the purpose of facilitating replication and further research, we have made our code and dataset publicly available at urlhttps://anonymous.4open.science/r/RAIT-9FB7/.