In plain words
Distillation transfers aspects of a teacher model’s behavior to a student model. The student is often smaller or cheaper to run, although smaller size is not a requirement.
A closer look
A teacher can provide probability distributions, example responses, labels, or reasoning demonstrations. The student learns from these targets using a training objective. The result is a new set of learned parameters; it is not simply a compressed archive of the teacher’s original weights.
Distillation is useful when a specialized workload does not require all of a large model’s abilities. Its success depends on the training examples, the student’s capacity, and the behavior being transferred. Teacher mistakes and biases can also be copied, so performance needs to be measured on fresh examples.
In practice
A team uses a capable model to label support messages, reviews the labels, and trains a smaller model to route incoming requests. The smaller model may serve that narrow task quickly without matching the teacher on general reasoning.
A useful distinction
A distilled model is not guaranteed to preserve every capability of its teacher. It can improve efficiency on selected tasks while losing breadth, robustness, or performance on unfamiliar inputs.