New Method Fixes On-Policy Distillation When Student Models Go Off Track
Researchers Kaixiang Zhao, Tianrun Yu, and Shangzhe Li published a paper identifying a core flaw in on-policy distillation (OPD), a technique where a student model learns from its own generated rollouts under token-level teacher supervision. OPD assumes the teacher can recover a correct answer from any prefix the student writes, a property the authors call 'recoverability' — but they show this fails when the student's rollout goes wrong. Their proposed fix, critique-guided on-policy distillation, uses the teacher's critique of the student's incorrect rollout to guide training instead of relying on the teacher to simply continue from a bad prefix. This matters because OPD is widely used to compress large language models into smaller deployable ones, and the recoverability failure means many distilled models are learning from corrupted supervision signals without anyone noticing.