Beyond Classification: Structured Supervision Aligns Visual Evidence with Medical Semantics

Graph Self-Supervision

Topological priors over patches / regions regularize the global embedding space.

Segmentation Co-Pretraining

Dense pixel-level constraints anchor features to lesion boundaries and foreground.

Image-Text Alignment

Cross-modal grounding links visual patches with clinical language and diagnostic logic.

Abstract

Vision Transformers (ViTs) have shown immense potential in medical image analysis. However, standard pre-training via global image classification suffers from spatial collapse, where models rely heavily on background shortcuts rather than localising critical foreground lesions.

To mitigate this limitation and align visual evidence with precise medical semantics, we systematically investigate alternative pre-training paradigms. Specifically, we evaluate three independent forms of structured supervision: topological priors via graph self-supervision, dense pixel-level constraints via segmentation, and cross-modal semantic grounding via image-text pairs.

Our empirical analysis reveals that while all three forms of structured supervision alleviate the global pooling bottleneck and steer visual attention towards foreground regions, image-text alignment achieves the strongest overall performance. When fine-tuned for downstream clinical classification, these models achieve superior accuracy and yield highly interpretable attention maps focused on true pathological features.

Structured Supervision Paradigms

We compare three independent forms of structured supervision that introduce spatial or semantic constraints beyond standard classification. All three alleviate spatial collapse to some extent, but reshape patch-to-global aggregation differently.

Overview of three structured pre-training paradigms.

Three structured pre-training paradigms beyond vanilla classification: graph self-supervision, segmentation co-pretraining, and image-text alignment.

Spatial Collapse in Medical ViTs

Driven by a global class-token objective, ViTs can learn shortcut associations with background textures, scanner artifacts, or anatomical context, instead of localising diagnostic lesions. We call this failure mode spatial collapse: the class token remains predictive, but its discriminative patch evidence is weakly aligned with lesion regions.

Across BUSI, HAM10000, and SIIM-ACR Pneumothorax, a standard ViT-B/16 reaches competitive accuracy (avg. ACC 0.836) but much weaker evidence localization (avg. PIB 0.168) than a convolutional reference (avg. PIB 0.685).

Qualitative comparison of ViT and ResNet evidence localization.

ViT vs ResNet-50 evidence localization. From left to right: original image, ground-truth lesion, patch-score heatmap, and top-scoring region.

Patch-score distribution used for evidence probing: discriminative evidence of vanilla ViTs often drifts away from lesion regions.

Dataset	ViT ACC	ViT PIB	ResNet ACC	ResNet PIB	ΔPIB
BUSI	0.888	0.071	0.862	0.690	+0.619
HAM10000	0.830	0.393	0.867	0.954	+0.561
SIIM-ACR	0.789	0.040	0.884	0.410	+0.370
Average	0.836	0.168	0.871	0.685	+0.517

High accuracy does not imply faithful lesion-level evidence alignment.

Results

Among the evaluated paradigms, image-text alignment provides the strongest overall evidence grounding: BiomedCLIP fine-tuning raises average PIB from 0.168 (vanilla ViT) to 0.410. Segmentation co-pretraining is especially helpful when lesions are small and spatially unstable (e.g., pneumothorax).

Model	Method	BUSI ACC / PIB	HAM10000 ACC / PIB	Pneumothorax ACC / PIB	Avg. PIB
ViT-B/16	Standard	0.888 / 0.071	0.830 / 0.393	0.847 / 0.040	0.168
BiomedCLIP	Fine-tuning	0.888 / 0.385	0.843 / 0.768	0.829 / 0.076	0.410
LVM-Med	Fine-tuning	0.715 / 0.114	0.836 / 0.248	0.864 / 0.051	0.138
ViT-B/16	Co-training	0.897 / 0.128	0.862 / 0.268	0.856 / 0.121	0.172

Patch-score heatmaps across evaluated methods on ISIC/HAM10000.

Patch-score heatmaps across methods. Image-text fine-tuning yields the most lesion-covering evidence maps on dermoscopic images.

Why Does Vanilla Classification Fail?

Image-level classification supervises only the final prediction from the global representation, without constraining which patches are aggregated into the class token. A foreground-background probe on BUSI shows that background-only class-token features remain class-separable (silhouette 0.631), while foreground-only features are much weaker (0.250). Shortcut regions can therefore support accurate labels without faithful lesion grounding.

(a) Original

(b) Background-only

(c) Foreground-only

Why Does Structured Supervision Help?

Structured pre-training strengthens coupling between local patch representations and the global diagnostic representation. On pneumothorax-positive samples, vanilla ViT produces weak patch evidence responses (mean 0.099), while structured methods shift the overall score distribution higher and improve foreground-background ranking (segmentation co-training reaches mean AUC 0.697).

Patch score distributions under different methods.

Patch evidence score distributions across methods.

Foreground–background AUC of patch evidence on pneumothorax.

Method	Mean AUC	Median AUC
ViT-B/16	0.287	0.249
ResNet50-L4	0.669	0.718
ViT Co-train	0.697	0.722
BiomedCLIP FT	0.431	0.420
LVM-Med FT	0.408	0.401

Structured methods improve foreground–background ranking of patch evidence.

Theoretical Analysis

Why do globally supervised ViTs prefer background shortcuts? We sketch the appendix analysis with a simplified single-layer linear attention model, showing that routing through homogeneous background patches is a high-SNR, easily accessible local minimum under SGD.

Linear attention + global pooling

With patches \(X=[x_1,\ldots,x_N]^\top\) and attention weights \(W=W_QW_K^\top\), a Softmax-free attention layer yields

\[ Z = X + XWX^\top X, \qquad \bar{z} = \bar{x} + \frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{N}(x_i^\top W x_j)\,x_j. \]

With head \(\theta\) and MSE loss \(\mathcal{L}=\tfrac12(\theta^\top\bar{z}-y^*)^2\), the gradient over \(W\) decomposes into competing foreground / background terms:

\[ \nabla_W\mathcal{L} = e\cdot\bar{x} \Bigg( \underbrace{\sum_{f\in\mathcal{F}}(\theta^\top x_f)x_f^\top}_{\Delta W_{fg}} + \underbrace{\sum_{b\in\mathcal{B}}(\theta^\top x_b)x_b^\top}_{\Delta W_{bg}} \Bigg). \]

SNR of the two paths

Treat foreground patches as high-variance semantic features \(x_f\sim\mathcal{N}(\mu_y,\Sigma_{fg})\) and background patches as nearly constant \(x_b\sim\mathcal{N}(c,\sigma_{bg}^2 I)\) with \(\sigma_{bg}\to 0\). Then

\[ \mathbb{E}[\Delta W_{bg}]\approx N_b(\theta^\top c)c^\top, \qquad \mathbb{V}[\Delta W_{bg}]\approx 0, \qquad \mathbb{V}[\Delta W_{fg}]\propto\mathrm{Tr}(\Sigma_{fg}^2). \] \[ \mathrm{SNR} = \frac{\|\mathbb{E}[\Delta W]\|}{\sqrt{\mathbb{V}[\Delta W]}} \;\Rightarrow\; \mathrm{SNR}_{bg}\to\infty \;\gg\; \mathrm{SNR}_{fg}. \]

Under SGD, background weights therefore grow like \(\mathcal{O}(t)\), whereas foreground weights behave like a noisy random walk \(\mathcal{O}(\sqrt{t})\).

Implication. Early training preferentially routes global information into homogeneous “register-like” background patches—the same mechanism behind spatial collapse. Structured supervision (graph / segmentation / image–text) raises the cost of this shortcut by forcing patch–global coupling to respect lesion evidence or clinical semantics.

Full derivations are in Appendix A of the paper.

BibTeX

@inproceedings{bai2026beyond,
  title     = {Beyond Classification: Structured Supervision Aligns Visual Evidence with Medical Semantics},
  author    = {Hexiang Bai and Hanyang Xu and Xiaoxue Li and Xiaoliang Wu and Shangde Gao and Hongxia Xu and Ke Liu},
  booktitle = {Proceedings of the British Machine Vision Conference ({BMVC})},
  year      = {2026},
  publisher = {BMVA}
}