# APOLLO: An Optimized Training Approach for Long-form Numerical Reasoning arXiv: https://arxiv.org/abs/2212.07249 ยท HTML: https://mlsanity.com/paper/2212.07249/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 31 of its 51 references that are in the map. - First posted on arXiv 14 Dec 2022. ## Builds on (most cited first, 30 of 31) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020) - [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017) - [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016) - [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018) - [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021) - [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022) - [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017) - [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019) - [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019) - [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019) - [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020) - [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021) - [ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering](https://mlsanity.com/paper/2210.03849/) (2022) - [Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context](https://mlsanity.com/paper/2005.00589/) (2020) - [Adversarial Retriever-Ranker for dense text retrieval](https://mlsanity.com/paper/2110.03611/) (2021) - [Poolingformer: Long Document Modeling with Pooling Attention](https://mlsanity.com/paper/2105.04371/) (2021) - [NumNet: Machine Reading Comprehension with Numerical Reasoning](https://mlsanity.com/paper/1910.06701/) (2019) - [Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction](https://mlsanity.com/paper/2203.10316/) (2022) - [Table-To-Text generation and pre-training with TabT5](https://mlsanity.com/paper/2210.09162/) (2022) - [ELASTIC: Numerical Reasoning with Adaptive Symbolic Compiler](https://mlsanity.com/paper/2210.10105/) (2022) - [DyRRen: A Dynamic Retriever-Reranker-Generator Model for Numerical Reasoning over Tabular and Textual Data](https://mlsanity.com/paper/2211.12668/) (2022) - [A Novel DeBERTa-based Model for Financial Question Answering Task](https://mlsanity.com/paper/2207.05875/) (2022) - [A Numerical Reasoning Question Answering System with Fine-grained Retriever and the Ensemble of Multiple Generators for FinQA](https://mlsanity.com/paper/2206.08506/) (2022) ## Built on by (most cited first, 8 of 8) - [StructLM: Towards Building Generalist Models for Structured Knowledge Grounding](https://mlsanity.com/paper/2402.16671/) (2024) - [ProTrix: Building Models for Planning and Reasoning over Tables with Sentence Context](https://mlsanity.com/paper/2403.02177/) (2024) - [SEER : A Knapsack approach to Exemplar Selection for In-Context HybridQA](https://mlsanity.com/paper/2310.06675/) (2023) - [FinQAPT: Empowering Financial Decisions with End-to-End LLM-driven Question Answering Pipeline](https://mlsanity.com/paper/2410.13959/) (2024) - [Retrieval-Augmented Chain-of-Thought in Semi-structured Domains](https://mlsanity.com/paper/2310.14435/) (2023) - [Program of Thoughts for Financial Reasoning: Leveraging Dynamic In-Context Examples and Generative Retrieval](https://mlsanity.com/paper/2510.13157/) (2025) - [Structure First, Reason Next: Enhancing a Large Language Model using Knowledge Graph for Numerical Reasoning in Financial Documents](https://mlsanity.com/paper/2601.07754/) (2026) - [Fighting Numerical Hallucinations via Data-centric Compilation for Online Financial QA](https://mlsanity.com/paper/2605.31064/) (2026) ## Abstract (arXiv) Long-form numerical reasoning in financial analysis aims to generate a reasoning program to calculate the correct answer for a given question. Previous work followed a retriever-generator framework, where the retriever selects key facts from a long-form document, and the generator generates a reasoning program based on retrieved facts. However, they treated all facts equally without considering the different contributions of facts with and without numbers. Meanwhile, the program consistency were ignored under supervised training, resulting in lower training accuracy and diversity. To solve these problems, we proposed APOLLO to improve the long-form numerical reasoning framework. For the retriever, we adopt a number-aware negative sampling strategy to enable the retriever to be more discriminative on key numerical facts. For the generator, we design consistency-based reinforcement learning and target program augmentation strategy based on the consistency of program execution results. Experimental results on the FinQA and ConvFinQA leaderboard verify the effectiveness of our proposed method, achieving the new state-of-the-art.