Structured Pruning for CLIP

Extended structured pruning to a multimodal dual-encoder for the first time, removing 40% of CLIP's parameters at a 1% accuracy cost. Built the sparse PyTorch training pipeline, decomposed Block Movement Pruning into modular steps, and added auxiliary losses to balance the two modalities.

2022 · Microsoft AI (internship) model compression · multimodal · efficiency
−40%
parameters removed from CLIP
−1%
accuracy, the only cost
first
structured pruning applied to a dual-encoder

The problem. Structured pruning (removing whole attention heads and FFN channels, so the model actually runs faster, not just smaller) was well studied for BERT-style encoders. Nobody had applied it to a contrastive dual-encoder like CLIP, where pruning one tower silently breaks its alignment with the other.

The idea. Prune both towers jointly under a shared budget, and treat the contrastive alignment as something to protect, not just the per-tower accuracy. I built a sparse training pipeline in PyTorch, decomposed Block Movement Pruning into modular steps so each could be scheduled and ablated, and introduced auxiliary losses that stabilize training and balance the two modalities while capacity is being removed.

Image encoder heads + FFN channels masked Text encoder heads + FFN channels masked Contrastive loss + alignment auxiliary losses Shared sparsity budget prune → recover → repeat mask updates flow back to both towers
Both towers are pruned under one budget, and the recovery schedule retrains on the contrastive objective so the towers stay aligned.

What I did. This was my internship project, and I did it end to end over three months: designed the method, built the sparse training pipeline, and ran the ablations over pruning schedules and auxiliary losses.