# Understand the Dynamic World: An End-to-End Knowledge Informed Framework for Open Domain Entity State Tracking

arXiv: https://arxiv.org/abs/2304.13854 · HTML: https://mlsanity.com/paper/2304.13854/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 38 references that are in the map.
- First posted on arXiv 26 Apr 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning](https://mlsanity.com/paper/2212.10773/) (2022)
- [CoLAKE: Contextualized Language and Knowledge Embedding](https://mlsanity.com/paper/2010.00309/) (2020)
- [Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension](https://mlsanity.com/paper/1805.06975/) (2018)
- [Modeling Naive Psychology of Characters in Simple Commonsense Stories](https://mlsanity.com/paper/1805.06533/) (2018)
- [Simulating Action Dynamics with Neural Process Networks](https://mlsanity.com/paper/1711.05313/) (2017)
- [The Materials Science Procedural Text Corpus: Annotating Materials Synthesis Procedures with Shallow Semantic Structures](https://mlsanity.com/paper/1905.06939/) (2019)
- [PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World](https://mlsanity.com/paper/2106.00188/) (2021)
- [A Dataset for Tracking Entities in Open Domain Procedural Text](https://mlsanity.com/paper/2011.08092/) (2020)
- [Revisiting Summarization Evaluation for Scientific Articles](https://mlsanity.com/paper/1604.00400/) (2016)
- [PLANET: Dynamic Content Planning in Autoregressive Transformers for Long-form Text Generation](https://mlsanity.com/paper/2203.09100/) (2022)
- [Minimize Exposure Bias of Seq2Seq Models in Joint Entity and Relation Extraction](https://mlsanity.com/paper/2009.07503/) (2020)
- [Time-Stamped Language Model: Teaching Language Models to Understand the Flow of Events](https://mlsanity.com/paper/2104.07635/) (2021)
- [Grape: Knowledge Graph Enhanced Passage Reader for Open-domain Question Answering](https://mlsanity.com/paper/2210.02933/) (2022)
- [Coalescing Global and Local Information for Procedural Text Understanding](https://mlsanity.com/paper/2208.12848/) (2022)
- [Semantic Structure based Query Graph Prediction for Question Answering over Knowledge Graph](https://mlsanity.com/paper/2204.10194/) (2022)
- [A Hierarchical N-Gram Framework for Zero-Shot Link Prediction](https://mlsanity.com/paper/2204.10293/) (2022)

## Built on by (most cited first, 9 of 9)

- [BiomedRAG: A Retrieval Augmented Large Language Model for Biomedicine](https://mlsanity.com/paper/2405.00465/) (2024)
- [SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2403.01599/) (2024)
- [How far is Language Model from 100% Few-shot Named Entity Recognition in Medical Domain](https://mlsanity.com/paper/2307.00186/) (2023)
- [CancerLLM: A Large Language Model in Cancer Domain](https://mlsanity.com/paper/2406.10459/) (2024)
- [A Condensed Transition Graph Framework for Zero-shot Link Prediction with Large Language Models](https://mlsanity.com/paper/2402.10779/) (2024)
- [GraPPI: A Retrieve-Divide-Solve GraphRAG Framework for Large-scale Protein-protein Interaction Exploration](https://mlsanity.com/paper/2501.16382/) (2025)
- [Efficient and Effective Internal Memory Retrieval for LLM-Based Healthcare Prediction](https://mlsanity.com/paper/2604.07659/) (2026)
- [RAGPPI: RAG Benchmark for Protein-Protein Interactions in Drug Discovery](https://mlsanity.com/paper/2505.23823/) (2025)
- [RIVO: Lightweight Adaptive Rerouting for Deep Research Agents](https://mlsanity.com/paper/2610.15910/) (2026)

## Abstract (arXiv)

Open domain entity state tracking aims to predict reasonable state changes of entities (i.e., [attribute] of [entity] was [before_state] and [after_state] afterwards) given the action descriptions. It's important to many reasoning tasks to support human everyday activities. However, it's challenging as the model needs to predict an arbitrary number of entity state changes caused by the action while most of the entities are implicitly relevant to the actions and their attributes as well as states are from open vocabularies. To tackle these challenges, we propose a novel end-to-end Knowledge Informed framework for open domain Entity State Tracking, namely KIEST, which explicitly retrieves the relevant entities and attributes from external knowledge graph (i.e., ConceptNet) and incorporates them to autoregressively generate all the entity state changes with a novel dynamic knowledge grained encoder-decoder framework. To enforce the logical coherence among the predicted entities, attributes, and states, we design a new constraint decoding strategy and employ a coherence reward to improve the decoding process. Experimental results show that our proposed KIEST framework significantly outperforms the strong baselines on the public benchmark dataset OpenPI.