Quantization, Distillation, and Deployment

What This Is

Training produces a model. Deployment requires a model that meets latency, memory, and energy budgets on the target hardware. The three primary tools for closing the gap are:

• quantization — reduce the numeric precision of weights and/or activations (fp32 → fp16, bf16, int8, int4)
• distillation — train a small student model to mimic a larger teacher, transferring behavior, not parameters
• pruning — remove weights or whole structural units (heads, channels) that carry little signal

This topic treats the three as a toolkit of deployment moves. Each changes inference cost, each changes accuracy, and each interacts with the others in predictable ways.

When You Use It

• your model works but is too slow, too large, or too expensive to run in production
• you need to fit an LLM on a consumer GPU (quantization to 4-bit is the standard move)
• you need to ship a fast mobile/edge classifier (distillation + int8 is the standard stack)
• you are serving the same model many times a second and 10% inference-latency improvement is worth weeks of engineering

Do Not Use It When

• accuracy is the current bottleneck, not latency — improve the model first, compress last
• you have not yet measured the real deployment cost — you are optimizing a guess
• the savings you need are enormous (10×+) — architecture change usually beats compression at that scale

The Three Primitives Side By Side

tool	what it changes	typical speedup	typical accuracy cost
quantization	numeric precision of weights / activations	2–4× (int8) up to 8× (int4 with careful kernels)	0–2% on most tasks; more on sensitive ones
distillation	model size and architecture	unbounded (depends on student size)	1–5% when done well; more if the gap is too large
pruning	number of non-zero parameters / structural units	1.5–3× with structured pruning; more with unstructured + sparse kernels	0–2% at moderate sparsity

Stack them when the deployment budget demands it. A common production path for mobile: distill to a smaller architecture, then quantize the student to int8.

Quantization

At the core: map a fp32 weight w to a lower-precision representation Q(w) such that the matmul can be done in cheaper arithmetic.

A symmetric linear quantizer:

w_q = round(w / s) · s,    s = max(|w|) / (2^(b-1) - 1)

b is the bit width (8 for int8; 4 for int4). Asymmetric variants shift by a zero-point offset; block-wise quantization uses a scale per small group of weights to preserve resolution near zero.

PTQ vs. QAT

Two routes to a quantized model:

• post-training quantization (PTQ) — take a trained fp32 model and quantize it, optionally using a small calibration set to fit the scales. Fast, no extra training, usually sufficient for int8.
• quantization-aware training (QAT) — simulate quantization during training (fake-quantize forward, straight-through backward) so the model's weights learn to be robust to the quantization error. Slower, required for extreme bit widths (int4 and below) or very small models where the accuracy drop under PTQ is unacceptable.

A practical rule:

• int8: try PTQ first; QAT only if PTQ loses more accuracy than you can afford
• int4: PTQ works for LLMs with GPTQ / AWQ / bitsandbytes; small non-LLM models often need QAT

QLoRA

Quantization meets fine-tuning: freeze the base model in 4-bit, add a LoRA adapter in bf16, train only the adapter. The base stays quantized throughout training, so memory footprint matches inference; the adapter carries the learning capacity. This is why you can fine-tune a 7B model on a single consumer GPU.

See PEFT and LoRA for the LoRA side of the story; QLoRA is the compression pairing of that topic.

Inspections That Matter

• weight distribution before quantization — heavy-tailed weight distributions quantize worse; outliers hurt more than the tails
• per-layer error — the first and last layers of a network are often most sensitive; sometimes they stay in higher precision
• calibration set size — PTQ scales are fit on a calibration set; too small a set produces fragile scales
• kernel availability — a bit-width that your framework cannot actually run with a fast kernel gives you only memory savings, not latency savings

Distillation

Instead of compressing the weights, train a smaller model to mimic the larger one. The student sees the teacher's outputs as additional supervision.

The canonical loss for classification (Hinton):

L = α · CE(y_student, y_true) + (1 - α) · T² · KL(soft_student, soft_teacher)

where soft_x = softmax(logits_x / T) and T > 1 is a temperature that smooths the teacher's probabilities. The smoothed distribution carries more information about similar classes than the one-hot label; that is the signal the student exploits.

Variants:

• response distillation — student matches teacher's output probabilities (Hinton)
• feature distillation — student matches intermediate activations (FitNets, layer alignment)
• attention distillation — for transformers, student matches attention maps (MiniLM, TinyBERT)

Distillation is the backbone of most "small but strong" models on Hugging Face (DistilBERT, TinyLlama, etc.).

Distillation Decisions

choice	guidance
student size	10–50% of teacher is the sweet spot; below 10% the gap usually cannot be closed
temperature T	2–5 for classification; higher for larger teachers
loss weight α	0.3–0.7; tune on validation
data	distillation on unlabeled data is powerful — the teacher provides pseudo-labels
layer alignment	helpful when student and teacher architectures are similar; finicky otherwise

The cheapest form — train the student on the teacher's soft predictions over a large unlabeled set — is often the strongest.

Pruning

Pruning removes weights (or whole structural components) deemed unimportant. Two families:

• unstructured pruning — zero out individual weights by magnitude. Models become sparse but need specialized sparse kernels for actual speedup.
• structured pruning — remove whole rows, channels, heads, or blocks. Real speedup on standard hardware.

The lottery-ticket hypothesis (Frankle & Carbin) changed the framing: in many networks, a small subset of weights at initialization suffices to reach full accuracy when re-trained. This suggests pruning is about finding which parts matter, not just shrinking.

Practical pruning today is dominated by structured pruning of transformer heads and MLP layers, especially in deployment pipelines for LLMs.

The Deployment Pipeline

A typical real-world compression pipeline for a transformer classifier:

trained fp32 model
        │
        ▼ distill to smaller architecture (optional)
smaller fp32 model
        │
        ▼ PTQ to int8 (with calibration)
int8 model
        │
        ▼ benchmark on target hardware, measure accuracy drop and latency
deployable artifact

For LLMs, the modern pipeline looks like:

trained fp16 or bf16 base
        │
        ▼ GPTQ / AWQ / bitsandbytes → int4 weights, fp16 activations
int4-weight model
        │
        ▼ (optional) QLoRA fine-tune on task data
adapted int4-base + bf16 adapter
        │
        ▼ serve with a kernel that supports mixed-precision matmul
deployable artifact

Benchmarking — The Honest Number

Compression benchmarks are famously misleading. A strong benchmark reports:

• accuracy on the actual target distribution, not a paper dataset
• latency measured on the target hardware, not desktop GPUs
• throughput at production batch size
• memory footprint including activation memory, not only weight memory
• energy per request if the deployment is power-constrained

A 4× size reduction on weights that produces a 1.1× wall-clock speedup is common and deeply unsatisfying. Measure latency.

What To Inspect

• accuracy-per-bit curve — for quantization, plot accuracy at fp32 / bf16 / int8 / int4; find the knee
• per-layer sensitivity — which layers cost the most accuracy when quantized; sometimes worth keeping them high-precision
• teacher-student agreement — after distillation, the rate at which the student and teacher agree on held-out data is a strong quality signal beyond accuracy
• calibration-set size effect — rerun PTQ with 100, 500, 2000 calibration samples; pick the smallest where scales stabilize
• sparsity vs. real latency — structured pruning with a real kernel; unstructured often wins only in theory

Failure Pattern

The canonical compression failure is accuracy looks fine in aggregate, a weak slice collapses. Quantization and distillation are both prone to degrading tail performance while the headline metric barely moves. A model that was 91% before compression and is 90% after might have dropped from 80% to 40% on the weak slice that actually matters. Always rerun slice-level evaluation after compression.

The second failure: optimization in vacuum. Teams achieve a 4× weight compression with 0.2% accuracy loss and ship. The target hardware does not support the kernel. Latency does not change. Measure end-to-end, not in isolation.

Common Mistakes

• quantizing without calibration data — scales default to naive max-based estimates and lose accuracy
• distilling a student 10× smaller than the teacher and expecting the gap to close
• evaluating compressed models on training-set-adjacent data instead of a realistic held-out set
• aggressive pruning followed by aggressive quantization without any retraining — compound errors
• shipping a quantized model without a latency measurement on target hardware
• ignoring the accuracy-per-slice effect of compression
• under-investing in the calibration set (PTQ quality is bounded by the calibration distribution)

Decision: Which Compression, In Which Order

A useful order of operations:

1. measure the real bottleneck — latency? memory? energy? each picks a different tool
2. distillation first if architecture flexibility exists — you gain the most headroom before precision tricks
3. quantization second — PTQ int8 is nearly free; int4 needs more care
4. structured pruning only if still over budget — diminishing returns; intricate engineering
5. QAT or further custom work only if the above are insufficient

Stack decisions:

goal	first tool	second tool
run LLM on consumer GPU	int4 weights (GPTQ/AWQ)	QLoRA for adaptation
fast mobile classifier	distillation	int8 quantization
cut cloud inference cost	distillation	int8 or bf16 quantization
run on CPU	distillation	int8 with a CPU-fast kernel

Practice

1. Take a trained ResNet-18 on CIFAR-10. Apply PTQ to int8 with 500 calibration images. Measure accuracy and model size before and after.
2. Apply QAT with fake-quantize during the last 5 epochs of training. Compare to PTQ.
3. Distill the same ResNet-18 into a smaller ResNet-10. Train the student with T = 4, α = 0.5. Report the agreement rate between student and teacher.
4. Benchmark latency of fp32, int8-PTQ, and distilled-int8 models on CPU using timeit. Report a wall-clock speedup, not a theoretical one.
5. Rerun the evaluation on a known weak slice (say, low-contrast images). Compare slice accuracies across all four configurations.
6. For an LLM angle: quantize a 1.3B parameter model with bitsandbytes to int4. Measure perplexity on WikiText-103 before and after. Note which layers, if any, lose the most.

Runnable Example

Compression work needs a real model and target hardware. Quantization: pip install bitsandbytes for LLMs, or PyTorch's native torch.ao.quantization for smaller models. Distillation: any HuggingFace Trainer recipe with a teacher forward pass. The browser runner is not the right home for these benchmarks — latency on a shared container is not the latency you will see in production.

Longer Connection

Compression is a deployment topic; it assumes a model already meets accuracy on the task. Adjacent topics:

• PEFT and LoRA — the QLoRA path ties quantization and fine-tuning into one pipeline
• Transfer and Fine-Tuning — distillation is a specialized fine-tune where the "labels" are the teacher's soft outputs
• Mixed Precision Training — bf16/fp16 training is the free first move; quantization pushes further at deployment time
• Self-Supervised and Representation Learning — many compressed models are distilled from SSL-pretrained teachers; the two topics compose

For the decision frame — "should I compress now or train better?" — the same rule as elsewhere in the academy applies: do not optimize what you have not measured. Baseline-First Task Solving still rules: prove the accuracy first, then compress for deployment.