Skip to content

Saliency And Explainability

When the model is frozen and you cannot peek inside with code, you can still ask the model to tell you where it is looking. Saliency maps are the answer. They also happen to be the workflow move when a task says "draw a rectangle that hides the most important region" or "explain why the model chose this class."

What This Is

Given a trained model f and an input x, produce a heatmap s(x) where high values mark the parts of x the model uses most for its decision. Four families you should know by name:

  • Vanilla gradient saliency. s = |∂L/∂x|. Fast, single backward pass. Noisy — almost any input pixel has some nonzero gradient.
  • Grad-CAM. For a CNN: take gradients at the last conv layer, global-average them per channel to get channel weights, weight the feature maps, ReLU, upsample. Produces a class-discriminative heatmap at feature-map resolution. The standard tool for "where did my ResNet look?"
  • Integrated Gradients. Integrate gradients along a straight line from a baseline (usually zero) to the input: IG = (x - x0) · ∫ ∂f/∂x dα. Satisfies the completeness axiom — attribution sums to f(x) - f(x0). Less noisy than vanilla gradients, more expensive (many forward/backward passes).
  • Attention rollout (for transformers). Multiply attention matrices across layers (with residual identity added), then take the row corresponding to the class token. Gives a token-wise importance map. No gradients needed.

There are many more (SmoothGrad, GradCAM++, SHAP, LIME). The four above cover 95% of what you will actually reach for.

When You Use It

  • the model is frozen and you need a decision about which input region matters
  • a task asks for a bounding box, mask, or rectangle that covers (or hides) the discriminative region
  • you are debugging a model that is right for the wrong reason
  • you need a sanity check that the model is not using a shortcut feature (background, watermark, demographic correlate)

Do Not Use It When

  • you need calibrated feature importance for a production decision — saliency maps are approximations, not causal attribution
  • the task is "explain the prediction to an end user in plain language" — saliency maps are for modelers, not laypeople
  • the model is not differentiable (pure text-based LLM behind an API) — different tools apply, usually prompt-level ablation

Grad-CAM In Practice

# last conv layer output: A with shape (C, H, W)
# target class logit: y_c
grads = torch.autograd.grad(y_c, A, retain_graph=True)[0]  # (C, H, W)
weights = grads.mean(dim=(1, 2))                            # (C,)
cam = (weights[:, None, None] * A).sum(dim=0)               # (H, W)
cam = F.relu(cam)
cam = cam / cam.max()
cam = F.interpolate(cam[None, None], size=x.shape[-2:], mode="bilinear").squeeze()

Two failure modes to watch: (1) pick the last conv layer, not any conv layer — earlier layers give low-level features, later ones give class-discriminative features. (2) the ReLU is non-optional; without it negative evidence pollutes the map.

Integrated Gradients In Practice

x0 = torch.zeros_like(x)           # baseline
steps = 50
alpha = torch.linspace(0, 1, steps)
interp = x0 + alpha[:, None, ...] * (x - x0)
grads = grad_fn(interp)            # batched gradient
avg_grad = grads.mean(dim=0)
ig = (x - x0) * avg_grad

The baseline choice matters. Zero-image is standard; alternatives are blurred input, mean pixel, or a learned baseline. Run with two baselines and compare.

Heatmap To Rectangle (Post-Processing)

When a task asks for one rectangle covering or hiding the important region:

  1. compute saliency s(x) (Grad-CAM or IG)
  2. for each candidate rectangle of the allowed shape and budget: score = sum of s inside it
  3. return the rectangle with the highest score

For large images, use integral-image tricks to make the candidate scan O(HW) instead of O(H²W²).

What To Inspect

  • heatmap for correct predictions vs incorrect predictions — they often look different; incorrect predictions often light up backgrounds
  • heatmap for a held-out class the model has never seen — should look near-uniform or random; if it lights up coherently, the method has a bias
  • sanity check by occlusion — mask out the top-saliency region and re-predict; confidence should drop sharply. If it does not, the saliency map is wrong.
  • saliency maps across several examples of the same class — they should be consistent; if each example lights up a different random patch, the method is noisy

Failure Pattern

  • vanilla-gradient saturation. The model is confident (softmax near 1), so the gradient of the output log-prob is near zero everywhere. The heatmap looks random. Fix: use the logit before softmax, or switch to Integrated Gradients.
  • Grad-CAM at the wrong layer. Middle-layer CAMs highlight textures, not semantics. Always use the final conv layer for classification tasks.
  • architecture mismatch. Grad-CAM assumes spatial feature maps. It does not work on pure MLPs or pooled-only heads. For transformers, use attention rollout instead.
  • heatmap = saliency, not = explanation. A rectangle around a dog's head does not prove the model used "dog head" features — it proves that region had high gradient. These are different claims.
  • one example is not a pattern. A single saliency map is an anecdote. Look at dozens before concluding anything about the model.

Quick Checks

  • does the method you chose match the architecture? (Grad-CAM → CNN, attention rollout → transformer, IG → any differentiable model)
  • are you using the logit before softmax, not the softmax probability?
  • did you sanity-check by masking the top-saliency region and confirming the prediction changed?
  • for Grad-CAM: are you at the last conv layer?
  • is the heatmap upsampled to input resolution before you act on it?

Practice

Run academy/labs/saliency-and-explainability/src/saliency_workflow.py:

  • trains a small CNN on a synthetic shape dataset where the class is determined by a small localized marker
  • computes vanilla-gradient, Grad-CAM, and Integrated Gradients saliency
  • scores each method by "pointing game" accuracy — does the argmax of the saliency map fall inside the true marker?
  • adds a "rectangle-from-heatmap" post-processing step and measures how much of the true discriminative region a budget-constrained rectangle can cover
  • compares a frozen pretrained model against the from-scratch model to show how method behavior changes

You should leave able to pick the right saliency method for an architecture, sanity-check it with occlusion, and convert a heatmap into a rectangle when that is what the task wants.

Longer Connection

Saliency is one of three families of explainability methods:

  • attribution (gradients, Grad-CAM, IG, SHAP, LIME) — which input features mattered?
  • mechanistic (circuits, probing, activation patching) — what internal computation is happening?
  • counterfactual (what minimal change to the input flips the decision?) — useful for both explainability and robustness analysis

Attribution is the fastest and most practical in contest settings. Mechanistic interpretability is a research frontier. Counterfactuals are useful for defense and robustness. When a task says "explain" or "pick the region", it almost always wants attribution.