3 stories tagged Self Distillation
Researchers developed a way for AI models to improve themselves by only keeping the best versions of their own work. This could lead to better code generators that actually produce working programs. The method forces the AI to run its own code and only learn from successful attempts.
Fine-tuning large language models can increase hallucinations by degrading pre-trained knowledge. Researchers propose a self-distillation method to mitigate this issue. This could reshape how we approach model training.
Researchers propose Self-Distillation Zero (SD-Zero), a method that improves training efficiency by converting binary rewards into dense supervision. This approach outperforms traditional reinforcement learning methods in verifiable settings.