Preference Distillation: Distilling Large Language Models with Teacher-Student Preference Pairs
Abstract
Large Language Models (LLMs) have exhibited impressive capabilities across diverse range of tasks, yet their enormous parameter spaces present challenges in resource-constrained environments. Knowledge distillation (KD) offers a viable solution by transferring expertise from large teacher models to compact student models. Traditional methods like KL divergence can falter due to the student model's restricted expressivity, and relying solely on single teacher outputs may result in poorly calibrated student models. In this work, we propose a novel LLM distillation approach that leverages teacher-student preference pairs, steering the student's focus towards understanding the relative quality of outputs instead of merely replicating teacher outputs. This method offers dual benefits: it addresses the limitations of student model expressivity and improves sequence ranking calibration, thereby facilitating a more efficient knowledge transfer from teacher to student models. Extensive experiments on sequence generation tasks validate the effectiveness of our approach.
