Skip to content

Saliency And Explainability

When the model is frozen and you cannot peek inside with code, you can still ask the model to tell you where it is looking. Saliency maps are the answer. They also happen to be the workflow move when a task says "draw a rectangle that hides the most important region" or "explain why the model chose this class."

What This Is

Given a trained model f and an input x, produce a heatmap s(x) where high values mark the parts of x the model uses most for its decision. Four families you should know by name:

  • Vanilla gradient saliency. s = |∂L/∂x|. Fast, single backward pass. Noisy — almost any input pixel has some nonzero gradient.
  • Grad-CAM. For a CNN: take gradients at the last conv layer, global-average them per channel to get channel weights, weight the feature maps, ReLU, upsample. Produces a class-discriminative heatmap at feature-map resolution. The standard tool for "where did my ResNet look?"
  • Integrated Gradients. Integrate gradients along a straight line from a baseline (usually zero) to the input: IG = (x - x0) · ∫ ∂f/∂x dα. Satisfies the completeness axiom — attribution sums to f(x) - f(x0). Less noisy than vanilla gradients, more expensive (many forward/backward passes).
  • Attention rollout (for transformers). Multiply attention matrices across layers (with residual identity added), then take the row corresponding to the class token. Gives a token-wise importance map. No gradients needed.

There are many more (SmoothGrad, GradCAM++, SHAP, LIME). The four above cover 95% of what you will actually reach for.

When You Use It

  • the model is frozen and you need a decision about which input region matters
  • a task asks for a bounding box, mask, or rectangle that covers (or hides) the discriminative region
  • you are debugging a model that is right for the wrong reason
  • you need a sanity check that the model is not using a shortcut feature (background, watermark, demographic correlate)

Do Not Use It When

  • you need calibrated feature importance for a production decision — saliency maps are approximations, not causal attribution
  • the task is "explain the prediction to an end user in plain language" — saliency maps are for modelers, not laypeople
  • the model is not differentiable (pure text-based LLM behind an API) — different tools apply, usually prompt-level ablation

Grad-CAM In Practice

# last conv layer output: A with shape (C, H, W)
# target class logit: y_c
grads = torch.autograd.grad(y_c, A, retain_graph=True)[0]  # (C, H, W)
weights = grads.mean(dim=(1, 2))                            # (C,)
cam = (weights[:, None, None] * A).sum(dim=0)               # (H, W)
cam = F.relu(cam)
cam = cam / cam.max()
cam = F.interpolate(cam[None, None], size=x.shape[-2:], mode="bilinear").squeeze()

Two failure modes to watch: (1) pick the last conv layer, not any conv layer — earlier layers give low-level features, later ones give class-discriminative features. (2) the ReLU is non-optional; without it negative evidence pollutes the map.

Integrated Gradients In Practice

x0 = torch.zeros_like(x)           # baseline
steps = 50
alpha = torch.linspace(0, 1, steps)
interp = x0 + alpha[:, None, ...] * (x - x0)
grads = grad_fn(interp)            # batched gradient
avg_grad = grads.mean(dim=0)
ig = (x - x0) * avg_grad

The baseline choice matters. Zero-image is standard; alternatives are blurred input, mean pixel, or a learned baseline. Run with two baselines and compare.

Heatmap To Rectangle (Post-Processing)

When a task asks for one rectangle covering or hiding the important region:

  1. compute saliency s(x) (Grad-CAM or IG)
  2. for each candidate rectangle of the allowed shape and budget: score = sum of s inside it
  3. return the rectangle with the highest score

For large images, use integral-image tricks to make the candidate scan O(HW) instead of O(H²W²).

What To Inspect

  • heatmap for correct predictions vs incorrect predictions — they often look different; incorrect predictions often light up backgrounds
  • heatmap for a class absent from the current image — should not consistently highlight the same region as the predicted class; repeated overlap can expose method or model bias
  • occlusion check — compare masking top-saliency regions with random or low-saliency masks, ideally across insertion/deletion curves. A weak change is not a verdict by itself: masking can create out-of-distribution inputs, and redundant evidence elsewhere can preserve the prediction.
  • saliency maps across several examples of the same class — they should be consistent; if each example lights up a different random patch, the method is noisy

Failure Pattern

  • vanilla-gradient saturation. The model is confident (softmax near 1), so the gradient of the output log-prob is near zero everywhere. The heatmap looks random. Fix: use the logit before softmax, or switch to Integrated Gradients.
  • Grad-CAM at the wrong layer. Earlier layers often emphasize local texture while later layers carry more task-level structure. Start with the final convolutional block for classification, then compare another layer if spatial resolution or architecture makes that choice uninformative.
  • architecture mismatch. Grad-CAM assumes spatial feature maps. It does not work on pure MLPs or pooled-only heads. For transformers, use attention rollout instead.
  • heatmap = saliency, not = explanation. A rectangle around a dog's head does not prove the model used "dog head" features — it proves that region had high gradient. These are different claims.
  • one example is not a pattern. A single saliency map is an anecdote. Look at dozens before concluding anything about the model.

Quick Checks

  • does the method you chose match the architecture? (Grad-CAM → CNN, attention rollout → transformer, IG → any differentiable model)
  • are you using the logit before softmax, not the softmax probability?
  • did you sanity-check by masking the top-saliency region and confirming the prediction changed?
  • for Grad-CAM: are you at the last conv layer?
  • is the heatmap upsampled to input resolution before you act on it?

Practice

Run .venv/bin/python labs/saliency-and-explainability/src/saliency_workflow.py:

  • trains a small CNN on a synthetic shape dataset where the class is determined by a small localized marker
  • computes vanilla-gradient, Grad-CAM, and Integrated Gradients saliency
  • scores each method by "pointing game" accuracy — does the argmax of the saliency map fall inside the true marker?
  • adds a "rectangle-from-heatmap" post-processing step and measures how much of the true discriminative region a budget-constrained rectangle can cover
  • compares a frozen pretrained model against the from-scratch model to show how method behavior changes

You should leave able to pick the right saliency method for an architecture, sanity-check it with occlusion, and convert a heatmap into a rectangle when that is what the task wants.

Runnable Example

From the repository root, run:

.venv/bin/python labs/saliency-and-explainability/src/saliency_workflow.py

Inspect the method comparison and the occlusion sanity check. A visually sharp heatmap is not evidence unless masking its highest-saliency region changes the prediction.

Longer Connection

Saliency is one of three families of explainability methods:

  • attribution (gradients, Grad-CAM, IG, SHAP, LIME) — which input features mattered?
  • mechanistic (circuits, probing, activation patching) — what internal computation is happening?
  • counterfactual (what minimal change to the input flips the decision?) — useful for both explainability and robustness analysis

Attribution is the fastest and most practical in contest settings. Mechanistic interpretability is a research frontier. Counterfactuals are useful for defense and robustness. When a task says "explain" or "pick the region", it almost always wants attribution.