Creating an AI Observer: Generative Semantic Workspaces

An experienced human Observer reading a document -- such as a crime report -- creates a succinct plot-like textit``Working Memory'' comprising different actors, their prototypical roles and states at any point, their evolution over time based on their interactions, and even a map of missing Semantic parts anticipating them in the future. textitAn equivalent AI Observer currently does not exist. We introduce the [G]enerative [S]emantic [W]orkspace (GSW) -- comprising an textit``Operator'' and a textit``Reconciler'' -- that leverages advancements in LLMs to create a generative-style Semantic framework, as opposed to a traditionally predefined set of lexicon labels. Given a text segment Cn that describes an ongoing situation, the textitOperator instantiates actor-centric Semantic maps (termed ``Workspace instance'' Wn). The textitReconciler resolves differences between Wn and a ``Working memory'' Mn^* to generate the updated Mn+1^*. GSW outperforms well-known baselines on several tasks (~ 94% vs. FST, GLEN, BertSRL - multi-sentence Semantics extraction, ~ 15% vs. NLI-BERT, ~ 35% vs. QA). By mirroring the real Observer, GSW provides the first step towards Spatial Computing assistants capable of understanding individual intentions and predicting future behavior.