Skip to main content

​
Model Distillation

Model distillation is a training approach in which outputs from a teacher model help create examples for a student model. The goal is to teach a narrower task to a model that may better fit the workflow’s cost, latency, or deployment constraints. Distillation does not copy all capabilities of the teacher. Results depend on the task, teacher prompts, data coverage, student capacity, training method, and evaluation quality.

​
Teacher and Student

  • The teacher generates or improves candidate answers for representative inputs.
  • Reviewers verify those candidates against source material and requirements.
  • The approved examples form part of the student’s fine-tuning dataset.
  • The student is evaluated on cases that were not used for training.
The teacher and student can be any models supported by the chosen training workflow. Avoid model names in a long-lived design because provider catalogs change.

​
Distillation Workflow

1

Define a Narrow Task

Specify the inputs, expected outputs, quality criteria, and boundaries. Distillation is easier to evaluate when the target is well defined.
2

Build Evaluation Cases

Create a held-out set from realistic use. Include common cases, edge cases, and cases requiring uncertainty or refusal.
3

Generate Candidates

Ask the teacher for outputs using the same context and rules expected in production. Generate varied inputs instead of repeated templates.
4

Review and Correct

Check facts, reasoning, safety, tone, and format. Remove or rewrite weak examples; do not assume teacher output is correct.
5

Train the Student

Format the approved examples for a supported fine-tuning workflow and train a version of the student model.
6

Compare Models

Compare the student, teacher, and unmodified student base model on the held-out set and on operational requirements.
7

Monitor

Review production failures, data drift, and provider changes. Retrain only when new evidence supports it.

​
Example Teacher Instruction

Treat the output as a candidate for review, not as approved training data.

​
Evaluation Design

Choose measures that match the task:
  • Exact schema or format compliance
  • Accuracy against approved sources
  • Correct tool or escalation decision
  • Human review scores using a written rubric
  • Failure rate on safety and refusal cases
  • Latency and measured cost in the intended environment
Do not describe quality as a percentage unless the scoring method and test set make that percentage meaningful. Do not assume a fixed cost saving; calculate it from current provider pricing and measured token or resource use.

​
When Distillation Is a Poor Fit

  • The task changes frequently
  • Answers depend mainly on current knowledge
  • A smaller model cannot support the required modality or tool behavior
  • You cannot review the generated examples
  • You do not have a representative evaluation set
  • Errors have consequences that the workflow cannot safely contain

​
Data Safety

Teacher prompts and generated datasets may be sent to model providers. Remove unnecessary personal data and secrets, and confirm that data handling is approved for every provider involved.

Fine-Tuning

Review dataset preparation and evaluation fundamentals.

Training Guide

Follow the available B-Bot Hub training workflow.