mlsanityData to 14 Oct 2026

onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction

arXiv 2609.24983 ↗First posted 21 Sep 2026cs.CL cs.HC cs.LGcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks also fits: Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

We present onPanda, an interactive tool for efficiently annotating LLM alignment data and agent trajectories. onPanda adopts token-level correction as its core interaction: while reading a model response, the annotator locates the first inappropriate token and either picks a substitute from the model's candidate tokens or types the correct text via free-form editing. The system then truncates everything after that position and continues generation from the corrected prefix, repeating this locate-correct-continue loop until a satisfactory response is obtained. This mechanism lets annotators precisely steer model outputs at low cost: a small controlled study suggests that onPanda reduces median annotation time by 52% over manual post-editing. Since the vast majority of tokens in the final response are generated by the model itself, the resulting data largely preserves the model's sampling distribution and is well suited for constructing on-policy SFT and preference data. Furthermore, the token-level corrections recorded during annotation provide fine-grained supervision with precise positions and naturally paired positive--negative samples. onPanda also connects to external tools and harnesses, enabling interactive trajectory annotation in realistic environments. In addition, we release Panda-CVL, a dataset annotated with onPanda, together with a benchmark for token-level correction.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 8 of its 19 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

8 of 19 references.

References · 8

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Let's Verify Step by Stepcited by 3,789
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  4. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  5. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  6. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  7. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 292
  8. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 108

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025The Realignment Problem: When Right becomes Wrong in LLMs
  2. 2025Is On-Policy Data always the Best Choice for Direct Preference Optimization-based LM Alignment?
  3. 2026SpecAlign: Efficient Specification-Grounded Alignment of Large Language Models via Synthetic Data
  4. 2025Dynamic Alignment for Collective Agency: Toward a Scalable Self-Improving Framework for Open-Ended LLM Alignment
  5. 2025Aligning LLMs on a Budget: Inference-Time Alignment with Heuristic Reward Models
  6. 2025Targeting Misalignment: A Conflict-Aware Framework for Reward-Model-based LLM Alignment
  7. 2025SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning
  8. 2026PolicyBank: Evolving Policy Understanding for LLM Agents