Multi-Modal Fusion¶

What This Is¶

Multi-modal fusion is about one decision:

should you combine modalities at all, and if so, should the combination happen early, late, or in the middle of the model

Fusion is only useful when the second modality adds complementary signal. If one modality already solves the task, fusion can add complexity without improving judgment. If the modalities cover the same signal in different dress, fusion is decoration, not performance.

When You Use It¶

the task has text plus tabular, image plus text, audio plus metadata, or similar mixed inputs
one modality alone has plateaued and you have evidence the other modality carries signal about the failures
the modalities plausibly carry different evidence (text for intent, image for content; sensor readings for state, metadata for context)
you are ready to compare fused models against single-modality baselines honestly

Do Not Use It When¶

the modalities carry redundant signal — the fused model trains harder without winning more
one modality is noisy and easy to game — it will drag the clean modality down
the downstream consumer needs interpretable attributions — fused models with cross-attention resist clean credit assignment
you have not yet built the best single-modality baselines — fusion is a tie-breaker, not a first move

Start With The Baseline Rule¶

Do not fuse first. Start with:

best single-modality baseline A
best single-modality baseline B
only then compare a fusion strategy

If you skip that order, you do not know whether fusion helped or merely added noise. See Baseline-First Task Solving.

Fusion Choice Table¶

Strategy	Where combination happens	Better first use	Main risk
early fusion	raw features or first layer	feature spaces already aligned and scale-controlled (e.g., numeric + numeric metadata)	one modality's scale dominates
late fusion	combine final predictions or probabilities	each modality already has a decent standalone model	misses deeper cross-modal interactions
intermediate (mid-) fusion	combine learned encoder outputs	modalities need to interact through a learned space	harder to debug; the fusion layer can fail silently
cross-attention fusion	tokens/regions from one modality attend to the other	strong per-modality encoders exist (CLIP, BERT, ViT)	computationally heavy; attention can collapse
contrastive alignment	learn a shared space where matched pairs are close	large paired data (CLIP-style)	requires huge paired sets to work well

Late Fusion — The First Honest Comparison¶

Late fusion is the minimum-viable first test:

# Late fusion by probability averaging
text_proba = text_model.predict_proba(X_text)[:, 1]
tab_proba  = tab_model.predict_proba(X_tab)[:, 1]
fused_proba = 0.5 * text_proba + 0.5 * tab_proba

More careful variants:

weighted average — fit the weight on a validation fold
logistic stack — train a small logistic regression on the per-modality scores
rank fusion — convert each modality's output to a rank and combine ranks; helps when probability scales differ

The point is not that averaging is always best. It is that it gives a clean first test of whether the second modality is adding anything at all. If late fusion does not move the needle, earlier/deeper fusion is unlikely to help either.

Early Fusion — Only When Scales Are Controlled¶

Early fusion concatenates features before a single model:

import numpy as np
from sklearn.preprocessing import StandardScaler

scaler_tab = StandardScaler().fit(X_tab)
scaler_img_emb = StandardScaler().fit(X_img_emb)  # e.g., penultimate-layer CNN features

X_fused = np.hstack([scaler_tab.transform(X_tab),
                     scaler_img_emb.transform(X_img_emb)])
model = LogisticRegression().fit(X_fused, y)

Rules that keep early fusion honest:

scale each modality separately before concatenation (StandardScaler, not MinMaxScaler unless you know the range is fixed)
keep the per-modality dimensionalities reasonable — a 5-dim tabular plus a 2048-dim image embedding is an embedding-fusion, not an early-fusion
watch for modality dominance: if one modality has 10× the dimension or variance, the model will learn from it alone

Cross-Attention Fusion¶

When both modalities already have strong encoders (ViT for image, BERT for text), the modern move is cross-attention: use one modality's tokens as queries, the other modality's tokens as keys/values.

# Sketch — image tokens attend to text tokens
# img_tok: (B, N_img, D), txt_tok: (B, N_txt, D)
fused = nn.MultiheadAttention(embed_dim=D, num_heads=8, batch_first=True)(
    query=img_tok, key=txt_tok, value=txt_tok
)[0]   # image tokens enriched with text context

Families:

VLMs (Flamingo, BLIP-2) — cross-attention on top of a frozen text decoder
late interaction for retrieval (ColBERT) — token-level similarity, not pooled
modality-fused transformers (ViLT, ALBEF) — one transformer over both streams

See Attention and Transformers for the mechanism.

Contrastive Alignment — CLIP-Style¶

Rather than fuse per-example, you can align two modality encoders in a shared space using contrastive learning:

for (image_i, text_i) in a batch of N pairs:
    maximize sim(E_img(image_i), E_txt(text_i))
    minimize sim(E_img(image_i), E_txt(text_j))  for j != i

The loss is symmetric InfoNCE. Once trained, the encoders are independently useful and match at inference time. CLIP is the canonical example; see Self-Supervised and Representation Learning.

Contrastive alignment is the right tool when:

you have many paired examples across modalities
you want to support zero-shot classification or retrieval
you can afford the compute to train both encoders

It is the wrong tool when pairs are scarce — a small paired set produces a brittle joint space.

Modality Collapse — The Quiet Fusion Failure¶

The fusion-specific pathology: during training, the model learns to ignore one modality entirely. Signals:

ablating one modality's input (zeros, random noise) leaves the prediction essentially unchanged
the fused model's accuracy matches a single-modality model's accuracy to within noise
gradient norm on one encoder's parameters is orders of magnitude smaller than on the other

Diagnostics:

# Modality-ablation inspection
with torch.no_grad():
    base_logits = model(x_img, x_txt)
    img_off = model(torch.zeros_like(x_img), x_txt)
    txt_off = model(x_img, torch.zeros_like(x_txt))

print("base:", base_logits.mean().item())
print("img masked:", img_off.mean().item())        # should change a lot
print("txt masked:", txt_off.mean().item())        # should also change

If zeroing a modality barely changes the output, fusion failed. Fixes include:

modality dropout — randomly zero one modality during training; forces the model to use both
per-modality auxiliary losses — each encoder has a head that must solve the task alone
balanced learning rates or encoder warm-up — a frozen strong encoder and a trained-from-scratch weak encoder is a recipe for collapse on the strong one

What To Inspect¶

per-modality baseline score, and the per-modality score floor (chance / constant)
fused score vs. each baseline — gain must exceed CV band
modality-ablation — zero one modality at inference; does the output change?
per-class or per-slice gain — where exactly did fusion help? If it is a single slice, maybe a rule beats the fusion
calibration — fused probabilities often need recalibration; probability averaging rarely yields calibrated outputs
training stability — per-modality loss curves and gradient norms
robustness — drop 30% of one modality's inputs at test; does the fused model degrade gracefully?

Failure Pattern¶

The canonical failure is adding a second modality because it sounds richer. That produces:

raw feature scales dominating each other
a noisy modality hurting a clean baseline
a fused system that is harder to explain without real gain
calibration loss when probabilities are combined carelessly

The modern failure is large cross-modal models where one modality is silently ignored — strong accuracy on paper, zero robustness to modality corruption. The ablation inspection above catches it.

Common Mistakes¶

fusing before building the strongest single-modality baselines
concatenating features without normalization or scale checks
evaluating fused and single-modality models on different splits
assuming a second modality always adds complementary signal
using intermediate fusion before a simple late-fusion comparison
training a fused model with a strong frozen encoder on one side and a tiny from-scratch encoder on the other — guaranteed collapse on the strong side
reporting fused-model gains without a modality-ablation check

A Good Fusion Note¶

After one comparison, the learner should be able to say:

what each modality contributed on its own
why fusion was tried
which strategy was used first (late → mid → cross-attention)
which slice improved, if any
whether the extra complexity is justified
whether the fused model still uses both modalities (ablation check)

Decision: Which Fusion First¶

Situation	Start with	Escalate to
two decent per-modality models	late fusion (probability average)	logistic stack, rank fusion
per-modality features aligned, low-dim	early fusion after per-modality scaling	gradient-boosted tree on the concatenated features
two strong encoders, rich paired data	cross-attention fusion	full VLM-style joint training
paired data plentiful, want zero-shot retrieval	contrastive alignment (CLIP-style)	a joint encoder with in-domain fine-tuning
one modality dominates, other is sparse	treat other as conditioning (FiLM)	learn where conditioning helps via ablations

Practice¶

Build the strongest baseline for each modality alone. Report both scores with CV bands.
Compare late fusion (simple average, weighted average, rank fusion, logistic stack) against the better baseline. Report the winner and the margin.
Decide whether early fusion is justified by the feature structure (dimensionality, variance). If yes, run it and show the gap to late.
Train a cross-attention fusion model between two small encoders. Ablate each modality at inference. Report both ablation drops.
Induce modality collapse deliberately by freezing the strong encoder and training only the weak one. Show the ablation report reveals the collapse.
Name one case where adding the second modality would probably hurt and defend the call in two sentences.

Runnable Example¶

Longer Connection¶

Multi-modal fusion sits next to:

Baseline-First Task Solving — the discipline that every per-modality baseline must pass before fusion begins
Vision and Text Encoders — when the fusion question depends on pretrained representations
Self-Supervised and Representation Learning — where CLIP-style contrastive alignment is trained
Attention and Transformers — the mechanism behind cross-attention fusion
Reliability Slices — per-slice gain analysis is the honest way to justify fusion

The fusion is not the point. The named failure in the single-modality baseline that fusion is meant to rescue is the point. If no such failure is named, fusion should not be built.