Structured Pruning for CLIP
Extended structured pruning to a multimodal dual-encoder for the first time, removing 40% of CLIP's parameters at a 1% accuracy cost. Built the sparse PyTorch training pipeline, decomposed Block Movement Pruning into modular steps, and added auxiliary losses to balance the two modalities.
The problem. Structured pruning (removing whole attention heads and FFN channels, so the model actually runs faster, not just smaller) was well studied for BERT-style encoders. Nobody had applied it to a contrastive dual-encoder like CLIP, where pruning one tower silently breaks its alignment with the other.
The idea. Prune both towers jointly under a shared budget, and treat the contrastive alignment as something to protect, not just the per-tower accuracy. I built a sparse training pipeline in PyTorch, decomposed Block Movement Pruning into modular steps so each could be scheduled and ablated, and introduced auxiliary losses that stabilize training and balance the two modalities while capacity is being removed.
What I did. This was my internship project, and I did it end to end over three months: designed the method, built the sparse training pipeline, and ran the ablations over pruning schedules and auxiliary losses.