Multi-Modal Fusion¶

What This Is¶

Multi-modal fusion is about one decision:

should you combine modalities at all, and if so, should the combination happen early, late, or in the middle of the model

Fusion is only useful when the second modality adds complementary signal. If one modality already solves the task, fusion can add complexity without improving judgment. If the modalities cover the same signal in different dress, fusion is decoration, not performance.

When You Use It¶

the task has text plus tabular, image plus text, audio plus metadata, or similar mixed inputs
one modality alone has plateaued and you have evidence the other modality carries signal about the failures
the modalities plausibly carry different evidence (text for intent, image for content; sensor readings for state, metadata for context)
you are ready to compare fused models against single-modality baselines honestly

Do Not Use It When¶

the modalities carry redundant signal — the fused model trains harder without winning more
one modality is noisy and easy to game — it will drag the clean modality down
the downstream consumer needs interpretable attributions — fused models with cross-attention resist clean credit assignment
you have not yet built the best single-modality baselines — fusion is a tie-breaker, not a first move

Start With The Baseline Rule¶

Do not fuse first. Start with:

best single-modality baseline A
best single-modality baseline B
only then compare a fusion strategy

If you skip that order, you do not know whether fusion helped or merely added noise. See Baseline-First Task Solving.

Fusion Choice Table¶

Strategy	Where combination happens	Better first use	Main risk
early fusion	raw features or first layer	feature spaces already aligned and scale-controlled (e.g., numeric + numeric metadata)	one modality's scale dominates
late fusion	combine final predictions or probabilities	each modality already has a decent standalone model	misses deeper cross-modal interactions
intermediate (mid-) fusion	combine learned encoder outputs	modalities need to interact through a learned space	harder to debug; the fusion layer can fail silently
cross-attention fusion	tokens/regions from one modality attend to the other	strong per-modality encoders exist (CLIP, BERT, ViT)	computationally heavy; attention can collapse
contrastive alignment	learn a shared space where matched pairs are close	large paired data (CLIP-style)	requires huge paired sets to work well

Late Fusion — The First Honest Comparison¶

Late fusion is the minimum-viable first test:

# Predictions on a final evaluation split not used to fit either model.
text_eval = text_model.predict_proba(X_text_eval)[:, 1]
tab_eval = tab_model.predict_proba(X_tab_eval)[:, 1]
fused_eval = 0.5 * text_eval + 0.5 * tab_eval

More careful variants:

weighted average — choose the weight from out-of-fold (OOF) training predictions, then evaluate once on a separate split
logistic stack — train a small logistic regression on OOF per-modality scores; in-sample base-model scores leak
rank fusion — convert each modality's output to a rank and combine ranks; helps when probability scales differ

The point is not that averaging is always best. It gives a clean first test of whether independently modeled modalities add complementary signal. A failed late-fusion test lowers confidence in fusion, but early or cross-modal fusion can still help when the useful signal lies specifically in interactions.

Early Fusion — Only When Scales Are Controlled¶

Early fusion concatenates features before a single model:

import numpy as np
from sklearn.preprocessing import StandardScaler

scaler_tab = StandardScaler().fit(X_tab_train)
scaler_img_emb = StandardScaler().fit(X_img_train)  # e.g., penultimate-layer CNN features

X_fused_train = np.hstack([scaler_tab.transform(X_tab_train),
                            scaler_img_emb.transform(X_img_train)])
X_fused_eval = np.hstack([scaler_tab.transform(X_tab_eval),
                           scaler_img_emb.transform(X_img_eval)])
model = LogisticRegression().fit(X_fused_train, y_train)
eval_proba = model.predict_proba(X_fused_eval)[:, 1]

Rules that keep early fusion honest:

scale each modality separately before concatenation (StandardScaler, not MinMaxScaler unless you know the range is fixed)
keep the per-modality dimensionalities and regularization reasonable; a 2048-dimensional embedding can dominate a 5-dimensional tabular block without careful preprocessing
watch for modality dominance: much larger dimension or variance can bias learning toward one modality, but does not guarantee that the other is ignored

Cross-Attention Fusion¶

When both modalities already have strong encoders (ViT for image, BERT for text), the modern move is cross-attention: use one modality's tokens as queries, the other modality's tokens as keys/values.

# Sketch — image tokens attend to text tokens
# img_tok: (B, N_img, D), txt_tok: (B, N_txt, D)
fused = nn.MultiheadAttention(embed_dim=D, num_heads=8, batch_first=True)(
    query=img_tok, key=txt_tok, value=txt_tok
)[0]   # image tokens enriched with text context

Families:

VLMs (for example, Flamingo and BLIP-2) — connect pretrained vision and language components with attention-based modules; their exact freezing and cross-attention designs differ
late interaction for retrieval (ColBERT) — token-level similarity, not pooled
modality-fused transformers (ViLT, ALBEF) — one transformer over both streams

See Attention and Transformers for the mechanism.

Contrastive Alignment — CLIP-Style¶

Rather than fuse per-example, you can align two modality encoders in a shared space using contrastive learning:

for (image_i, text_i) in a batch of N pairs:
    maximize sim(E_img(image_i), E_txt(text_i))
    minimize sim(E_img(image_i), E_txt(text_j))  for j != i

The loss is symmetric InfoNCE. Once trained, the encoders are independently useful and match at inference time. CLIP is the canonical example; see Self-Supervised and Representation Learning.

Contrastive alignment is the right tool when:

you have many paired examples across modalities
you want to support zero-shot classification or retrieval
you can afford the compute to train or adapt the encoders

Learning both encoders from scratch is difficult when pairs are scarce. Adapting pretrained encoders on a small paired set is a separate, often more practical experiment.

Modality Collapse — The Quiet Fusion Failure¶

The fusion-specific pathology: during training, the model learns to ignore one modality entirely. Signals:

ablating one modality's input (zeros, random noise) leaves the prediction essentially unchanged
the fused model's accuracy matches a single-modality model's accuracy to within noise
gradient norm on one encoder's parameters is orders of magnitude smaller than on the other

Diagnostics:

# Modality-ablation inspection
with torch.no_grad():
    base_logits = model(x_img, x_txt)
    img_off = model(torch.zeros_like(x_img), x_txt)
    txt_off = model(x_img, torch.zeros_like(x_txt))

print("base:", base_logits.mean().item())
print("img masked:", img_off.mean().item())        # should change a lot
print("txt masked:", txt_off.mean().item())        # should also change

Zeroing can be far outside the training distribution, so treat this as a stress test rather than a causal proof. Pair it with retraining/validation without that modality or with realistic missing-modality examples. If all tests show no useful contribution, fusion has failed. Possible interventions include:

modality dropout — randomly zero one modality during training; forces the model to use both
per-modality auxiliary losses — each encoder has a head that must solve the task alone
balanced learning rates or encoder warm-up — large differences in encoder quality or optimization speed can encourage dominance

What To Inspect¶

per-modality baseline score, and the per-modality score floor (chance / constant)
paired fused-vs-baseline deltas on identical folds or examples, with uncertainty from repeated CV or a paired bootstrap; fold standard deviation alone is not a confidence interval for the gain
modality-ablation — compare retraining without a modality and realistic missing-modality stress tests; does useful held-out behavior change?
per-class or per-slice gain — where exactly did fusion help? If it is a single slice, maybe a rule beats the fusion
calibration — check fused probabilities explicitly; averaging calibrated inputs does not guarantee a calibrated result
training stability — per-modality loss curves and gradient norms
robustness — drop 30% of one modality's inputs at test; does the fused model degrade gracefully?

Failure Pattern¶

The canonical failure is adding a second modality because it sounds richer. That produces:

raw feature scales dominating each other
a noisy modality hurting a clean baseline
a fused system that is harder to explain without real gain
calibration loss when probabilities are combined carelessly

The modern failure is large cross-modal models where one modality is silently ignored — strong accuracy on paper, zero robustness to modality corruption. The ablation inspection above catches it.

Common Mistakes¶

fusing before building the strongest single-modality baselines
concatenating features without normalization or scale checks
evaluating fused and single-modality models on different splits
assuming a second modality always adds complementary signal
using intermediate fusion before a simple late-fusion comparison
training encoders with very different starting quality or learning dynamics without checking for modality dominance
reporting fused-model gains without a modality-ablation check

A Good Fusion Note¶

After one comparison, the learner should be able to say:

what each modality contributed on its own
why fusion was tried
which strategy was used first (late → mid → cross-attention)
which slice improved, if any
whether the extra complexity is justified
whether the fused model still uses both modalities (ablation check)

Decision: Which Fusion First¶

Situation	Start with	Escalate to
two decent per-modality models	late fusion (probability average)	logistic stack, rank fusion
per-modality features aligned, low-dim	early fusion after per-modality scaling	gradient-boosted tree on the concatenated features
two strong encoders, rich paired data	cross-attention fusion	full VLM-style joint training
paired data plentiful, want zero-shot retrieval	contrastive alignment (CLIP-style)	a joint encoder with in-domain fine-tuning
one modality dominates, other is sparse	treat other as conditioning (FiLM)	learn where conditioning helps via ablations

Practice¶

Build the strongest baseline for each modality alone. Report both scores with CV bands.
Compare late fusion (simple average, weighted average, rank fusion, logistic stack) against the better baseline. Report the winner and the margin.
Decide whether early fusion is justified by the feature structure (dimensionality, variance). If yes, run it and show the gap to late.
Train a cross-attention fusion model between two small encoders. Ablate each modality at inference. Report both ablation drops.
Create an imbalanced setup with one strong encoder and one weak encoder. Test whether collapse occurs; compare zeroing, realistic corruption, and retrain-without-modality diagnostics.
Name one case where adding the second modality would probably hurt and defend the call in two sentences.

Runnable Example¶

Longer Connection¶

Multi-modal fusion sits next to:

Baseline-First Task Solving — the discipline that every per-modality baseline must pass before fusion begins
Vision and Text Encoders — when the fusion question depends on pretrained representations
Self-Supervised and Representation Learning — where CLIP-style contrastive alignment is trained
Attention and Transformers — the mechanism behind cross-attention fusion
Reliability Slices — per-slice gain analysis is the honest way to justify fusion

The fusion is not the point. The named failure in the single-modality baseline that fusion is meant to rescue is the point. If no such failure is named, fusion should not be built.