Quantization, Distillation, and Deployment¶

What This Is¶

Training produces a model. Deployment requires a model that meets latency, memory, and energy budgets on the target hardware. The three primary tools for closing the gap are:

quantization — reduce the numeric precision of weights and/or activations (fp32 → fp16, bf16, int8, int4)
distillation — train a small student model to mimic a larger teacher, transferring behavior, not parameters
pruning — remove weights or whole structural units (heads, channels) that carry little signal

This topic treats the three as a toolkit of deployment moves. Each changes inference cost, each changes accuracy, and each interacts with the others in predictable ways.

When You Use It¶

your model works but is too slow, too large, or too expensive to run in production
you need to fit an LLM on a consumer GPU (quantization to 4-bit is the standard move)
you need to ship a fast mobile/edge classifier (distillation + int8 is the standard stack)
you are serving the same model many times a second and 10% inference-latency improvement is worth weeks of engineering

Do Not Use It When¶

accuracy is the current bottleneck, not latency — improve the model first, compress last
you have not yet measured the real deployment cost — you are optimizing a guess
the savings you need are enormous (10×+) — architecture change usually beats compression at that scale

The Three Primitives Side By Side¶

tool	what it changes	typical speedup	typical accuracy cost
quantization	numeric precision of weights / activations	2–4× (int8) up to 8× (int4 with careful kernels)	0–2% on most tasks; more on sensitive ones
distillation	model size and architecture	unbounded (depends on student size)	1–5% when done well; more if the gap is too large
pruning	number of non-zero parameters / structural units	1.5–3× with structured pruning; more with unstructured + sparse kernels	0–2% at moderate sparsity

Stack them when the deployment budget demands it. A common production path for mobile: distill to a smaller architecture, then quantize the student to int8.

Quantization¶

At the core: map a fp32 weight w to a lower-precision representation Q(w) such that the matmul can be done in cheaper arithmetic.

A symmetric linear quantizer:

w_q = round(w / s) · s,    s = max(|w|) / (2^(b-1) - 1)

b is the bit width (8 for int8; 4 for int4). Asymmetric variants shift by a zero-point offset; block-wise quantization uses a scale per small group of weights to preserve resolution near zero.

PTQ vs. QAT¶

Two routes to a quantized model:

post-training quantization (PTQ) — take a trained fp32 model and quantize it, optionally using a small calibration set to fit the scales. Fast, no extra training, usually sufficient for int8.
quantization-aware training (QAT) — simulate quantization during training (fake-quantize forward, straight-through backward) so the model can adapt to quantization error. It costs extra training and is useful when PTQ misses the accuracy target; it is not inherently required for every int4 model.

A practical rule:

int8: try PTQ first; QAT only if PTQ loses more accuracy than you can afford
int4: PTQ works for LLMs with GPTQ / AWQ / bitsandbytes; small non-LLM models often need QAT

QLoRA¶

Quantization meets fine-tuning: freeze a quantized 4-bit base, add a higher-precision LoRA adapter, and train only the adapter. Training still needs adapter parameters, optimizer states, activations, and temporary dequantized compute values, so its memory footprint exceeds inference. QLoRA can greatly reduce fine-tuning memory, but whether a given model fits depends on sequence length, batch size, checkpointing, optimizer, and hardware.

See PEFT and LoRA for the LoRA side of the story; QLoRA is the compression pairing of that topic.

Inspections That Matter¶

weight distribution before quantization — heavy-tailed weight distributions quantize worse; outliers hurt more than the tails
per-layer error — the first and last layers of a network are often most sensitive; sometimes they stay in higher precision
calibration set size — PTQ scales are fit on a calibration set; too small a set produces fragile scales
kernel availability — a bit-width that your framework cannot actually run with a fast kernel gives you only memory savings, not latency savings

Distillation¶

Instead of compressing the weights, train a smaller model to mimic the larger one. The student sees the teacher's outputs as additional supervision.

The canonical loss for classification (Hinton):

L = α · CE(y_student, y_true) + (1 - α) · T² · D_KL(soft_teacher || soft_student)

where soft_x = softmax(logits_x / T) and T > 1 is a temperature that smooths the teacher's probabilities. The smoothed distribution carries more information about similar classes than the one-hot label; that is the signal the student exploits.

Variants:

response distillation — student matches teacher's output probabilities (Hinton)
feature distillation — student matches intermediate activations (FitNets, layer alignment)
attention distillation — for transformers, student matches attention maps (MiniLM, TinyBERT)

DistilBERT is an example of a distilled model. Small size alone does not imply distillation: TinyLlama was pretrained on a large token corpus.

The formula uses mathematical KL ordering. In PyTorch, F.kl_div(log_softmax(student_logits / T, dim=-1), softmax(teacher_logits / T, dim=-1), reduction="batchmean") implements this direction because the first argument contains student log-probabilities and the second contains teacher target probabilities. Detach the teacher targets and multiply the soft loss by T**2. See the original distillation paper.

Distillation Decisions¶

choice	guidance
student size	10–50% of teacher is the sweet spot; below 10% the gap usually cannot be closed
temperature `T`	2–5 for classification; higher for larger teachers
loss weight `α`	0.3–0.7; tune on validation
data	distillation on unlabeled data is powerful — the teacher provides pseudo-labels
layer alignment	helpful when student and teacher architectures are similar; finicky otherwise

The cheapest form — train the student on the teacher's soft predictions over a large unlabeled set — is often the strongest.

Pruning¶

Pruning removes weights (or whole structural components) deemed unimportant. Two families:

unstructured pruning — zero out individual weights by magnitude. Models become sparse but need specialized sparse kernels for actual speedup.
structured pruning — remove whole rows, channels, heads, or blocks. Real speedup on standard hardware.

The lottery-ticket hypothesis reports that, for studied networks and training setups, some sparse subnetworks identified from an initialization can train to match the original network's test accuracy. It motivates research on sparse trainable subnetworks, but does not imply that arbitrary models contain an easily recoverable full-accuracy ticket.

Practical pruning today is dominated by structured pruning of transformer heads and MLP layers, especially in deployment pipelines for LLMs.

The Deployment Pipeline¶

A typical real-world compression pipeline for a transformer classifier:

trained fp32 model
        │
        ▼ distill to smaller architecture (optional)
smaller fp32 model
        │
        ▼ PTQ to int8 (with calibration)
int8 model
        │
        ▼ benchmark on target hardware, measure accuracy drop and latency
deployable artifact

For LLMs, the modern pipeline looks like:

trained fp16 or bf16 base
        │
        ▼ GPTQ / AWQ / bitsandbytes → int4 weights, fp16 activations
int4-weight model
        │
        ▼ (optional) QLoRA fine-tune on task data
adapted int4-base + bf16 adapter
        │
        ▼ serve with a kernel that supports mixed-precision matmul
deployable artifact

Benchmarking — The Honest Number¶

Compression benchmarks are famously misleading. A strong benchmark reports:

accuracy on the actual target distribution, not a paper dataset
latency measured on the target hardware, not desktop GPUs
throughput at production batch size
memory footprint including activation memory, not only weight memory
energy per request if the deployment is power-constrained

A 4× size reduction on weights that produces a 1.1× wall-clock speedup is common and deeply unsatisfying. Measure latency.

What To Inspect¶

accuracy-per-bit curve — for quantization, plot accuracy at fp32 / bf16 / int8 / int4; find the knee
per-layer sensitivity — which layers cost the most accuracy when quantized; sometimes worth keeping them high-precision
teacher-student agreement — after distillation, the rate at which the student and teacher agree on held-out data is a strong quality signal beyond accuracy
calibration-set size effect — rerun PTQ with 100, 500, 2000 calibration samples; pick the smallest where scales stabilize
sparsity vs. real latency — structured pruning with a real kernel; unstructured often wins only in theory

Failure Pattern¶

The canonical compression failure is accuracy looks fine in aggregate, a weak slice collapses. Quantization and distillation are both prone to degrading tail performance while the headline metric barely moves. A model that was 91% before compression and is 90% after might have dropped from 80% to 40% on the weak slice that actually matters. Always rerun slice-level evaluation after compression.

The second failure: optimization in vacuum. Teams achieve a 4× weight compression with 0.2% accuracy loss and ship. The target hardware does not support the kernel. Latency does not change. Measure end-to-end, not in isolation.

Common Mistakes¶

quantizing without calibration data — scales default to naive max-based estimates and lose accuracy
distilling a student 10× smaller than the teacher and expecting the gap to close
evaluating compressed models on training-set-adjacent data instead of a realistic held-out set
aggressive pruning followed by aggressive quantization without any retraining — compound errors
shipping a quantized model without a latency measurement on target hardware
ignoring the accuracy-per-slice effect of compression
under-investing in the calibration set (PTQ quality is bounded by the calibration distribution)

Decision: Which Compression, In Which Order¶

One reasonable starting order, to validate empirically for the target model and hardware:

measure the real bottleneck — latency? memory? energy? each picks a different tool
test the cheapest compatible change — often PTQ on supported hardware, or distillation when architecture size is the bottleneck
combine methods only after measuring interactions — distillation, pruning, and quantization can change one another's accuracy and latency
use structured pruning when supported shapes and kernels produce a measured gain
QAT or further custom work only if the above are insufficient

Stack decisions:

goal	first tool	second tool
run LLM on consumer GPU	int4 weights (GPTQ/AWQ)	QLoRA for adaptation
fast mobile classifier	distillation	int8 quantization
cut cloud inference cost	distillation	int8 or bf16 quantization
run on CPU	distillation	int8 with a CPU-fast kernel

Practice¶

Take a trained ResNet-18 on CIFAR-10. Apply PTQ to int8 with 500 calibration images. Measure accuracy and model size before and after.
Apply QAT with fake-quantize during the last 5 epochs of training. Compare to PTQ.
Distill the same ResNet-18 into a smaller ResNet-10. Train the student with T = 4, α = 0.5. Report the agreement rate between student and teacher.
Benchmark latency of fp32, int8-PTQ, and distilled-int8 models on CPU using timeit. Report a wall-clock speedup, not a theoretical one.
Rerun the evaluation on a known weak slice (say, low-contrast images). Compare slice accuracies across all four configurations.
For an LLM angle: quantize a 1.3B parameter model with bitsandbytes to int4. Measure perplexity on WikiText-103 before and after. Note which layers, if any, lose the most.

Runnable Example¶

Run the offline distillation fixture from the repository root:

.venv/bin/python examples/deep-learning-recipes/advanced_mechanics_demo.py --demo compression

Inspect teacher/student parameter counts and distillation error before and after training. The example validates mechanics only; deployment still requires latency and artifact-size measurements on the actual target hardware.

Longer Connection¶

Compression is a deployment topic; it assumes a model already meets accuracy on the task. Adjacent topics:

PEFT and LoRA — the QLoRA path ties quantization and fine-tuning into one pipeline
Transfer and Fine-Tuning — distillation is a specialized fine-tune where the "labels" are the teacher's soft outputs
Mixed Precision Training — bf16/fp16 training is the free first move; quantization pushes further at deployment time
Self-Supervised and Representation Learning — many compressed models are distilled from SSL-pretrained teachers; the two topics compose

For the decision frame — "should I compress now or train better?" — the same rule as elsewhere in the academy applies: do not optimize what you have not measured. Baseline-First Task Solving still rules: prove the accuracy first, then compress for deployment.