LearnAI

Navigation

Introduction to AI

Machine Learning

Deep Learning

NLP & LLMs

Computer Vision

Generative AI

Reinforcement Learning

AI Ethics & Governance

Tools & Frameworks

General

Cross-Modal Contrastive Learning

Cross-modal contrastive learning trains neural networks to align representations across different modalities — most prominently vision and language — by bringing paired samples (an image and its caption) close together in a shared embedding space while pushing apart unpaired samples. The resulting models learn rich, semantically meaningful representations without manual class labels, enabling zero-shot generalization to tasks and categories never seen during training.

The Contrastive Alignment Objective

Given a batch of $N$ image-text pairs ${(I_i, T_i)}_{i=1}^N$, a vision encoder $f_v$ and a text encoder $f_t$ produce normalized embeddings:

$$z_i^v = \frac{f_v(I_i)}{|f_v(I_i)|_2}, \qquad z_i^t = \frac{f_t(T_i)}{|f_t(T_i)|_2}$$

The InfoNCE loss (also called NT-Xent or CLIP loss) treats each pair $(I_i, T_i)$ as a positive and all $N-1$ other texts as negatives for image $I_i$, and vice versa:

$$\mathcal{L}\text{CLIP} = -\frac{1}{2N}\sum{i=1}^N \left[ \log \frac{\exp(z_i^v \cdot z_i^t / \tau)}{\sum_j \exp(z_i^v \cdot z_j^t / \tau)} + \log \frac{\exp(z_i^t \cdot z_i^v / \tau)}{\sum_j \exp(z_i^t \cdot z_j^v / \tau)} \right]$$

where $\tau$ is a learned temperature parameter. The loss encourages the cosine similarity of the matching pair to exceed all non-matching similarities — building a $N \times N$ similarity matrix where the diagonal is maximized.

Batch size is critical: with $N = 32{,}768$, each image has 32,767 in-batch negative texts. Larger batches provide harder negatives and stronger training signal, motivating the massive-scale training of CLIP and ALIGN.

CLIP: Contrastive Language-Image Pre-training

CLIP (Radford et al., OpenAI, 2021) established the modern paradigm for cross-modal contrastive learning:

  • Scale: trained on 400 million image-text pairs scraped from the internet (WIT-400M).
  • Architecture: ViT or ResNet vision encoder + Transformer text encoder, both producing 512- or 1024-dimensional embeddings.
  • Zero-shot classification: to classify an image into $C$ categories, compute cosine similarity between the image embedding and embeddings of $C$ text prompts (“a photo of a {class}”), then take the highest-similarity class. CLIP achieves 76.2% zero-shot accuracy on ImageNet, matching supervised ResNet-50 trained on 1.28M ImageNet labels.

The key insight of CLIP is that the web naturally contains paired supervision: images posted online typically have associated text (captions, alt text, surrounding descriptions) that describes image content. At 400M scale, this noisy paired supervision suffices for powerful generalization.

Prompt Engineering for CLIP

Zero-shot performance depends strongly on the text prompt format. “a photo of a {class}” consistently outperforms just “{class}”, because the prompt matches the distribution of training captions (which describe photos). Ensembling multiple prompts (“a photo of a {class}”, “a photo of the {class}”, “a high quality photo of a {class}”) further improves accuracy by averaging over prompt variations.

ALIGN: Scaling Beyond CLIP

ALIGN (Jia et al., Google, 2021) demonstrated that scale compensates for noise:

  • Training data: 1.8 billion noisy image-text pairs from the web — 4.5× more than CLIP, with minimal filtering.
  • Architecture: EfficientNet-L2 vision encoder + BERT text encoder.
  • Finding: at 1.8B pairs, ALIGN matches or exceeds CLIP despite much noisier data — indicating that data quantity can substitute for quality at sufficient scale.

ALIGN achieves 85.5% zero-shot accuracy on ImageNet with linear probing, demonstrating that the learned representations are highly transfer-capable even when the raw zero-shot accuracy (without prompt tuning) lags CLIP.

SigLIP: Sigmoid Loss for Efficiency

SigLIP (Zhai et al., Google, 2023) replaces the softmax contrastive loss with a sigmoid loss that treats each image-text pair independently as a binary classification:

$$\mathcal{L}\text{SigLIP} = -\frac{1}{N^2}\sum{i,j} \log \sigma!\left( y_{ij} \cdot (z_i^v \cdot z_j^t - b) \right)$$

where $y_{ij} = +1$ for matching pairs and $-1$ for non-matching pairs, and $b$ is a learned bias. Unlike softmax, the sigmoid loss does not require normalizing over the full batch — each pair’s loss is computed independently.

Advantages of SigLIP

  • No all-gather required: in distributed training, the softmax loss requires gathering all embeddings across all devices to compute the denominator. SigLIP only needs local pairs — each device computes its own pairs’ losses independently, enabling much more efficient large-scale training.
  • Better small-batch performance: when batch size is small, softmax contrastive learning degrades severely (few negatives). SigLIP maintains reasonable performance even with small batches because each pair is independently assessed.
  • Higher accuracy: SigLIP achieves better zero-shot and few-shot performance than CLIP at equivalent training compute.

CoCa: Contrastive Captioning

CoCa (Yu et al., Google, 2022) combines contrastive learning with generative learning in a single model:

  • A contrastive loss (CLIP-style) aligns the global image and text embeddings.
  • A captioning loss (autoregressive cross-entropy) trains the text decoder to generate captions conditioned on image features via cross-attention.

This dual objective produces a model that is simultaneously:

  • A strong retrieval model (via the contrastive representation).
  • A strong captioning and VQA model (via the generative decoder).

CoCa achieves state-of-the-art results on image captioning, VQA, and image classification by fine-tuning from the pre-trained checkpoint — demonstrating that contrastive and generative objectives are complementary rather than competing.

data2vec: Universal Self-Supervised Learning

data2vec (Baevski et al., Meta AI, 2022) extends cross-modal contrastive ideas to self-supervised learning across arbitrary modalities without requiring paired data:

  • For each modality (vision, text, speech), a masked version of the input is fed to a student encoder, and the target is the representation produced by a teacher network (exponential moving average of the student) from the unmasked input.
  • The same training objective applies regardless of modality — only the tokenizer (patch embeddings for images, BPE tokens for text, waveform segments for speech) changes.

data2vec achieves competitive performance with modality-specific self-supervised methods (MAE for vision, BERT for text, wav2vec 2.0 for speech) using a single training framework.

Negative Mining Strategies

The quality of negatives determines how much the model is pushed to learn fine-grained distinctions. Standard random in-batch negatives are often easy — most random text-image pairs are obviously unrelated.

Hard Negative Mining

Hard negatives are pairs that are semantically related but not matching — an image of a golden retriever paired with “a brown dog sitting on grass” (wrong dog, right scene). Training on hard negatives forces the model to encode fine-grained visual and linguistic distinctions.

  • HNSWLIB-based retrieval: after each epoch, find the most similar non-matching pairs using approximate nearest neighbor search and add them as explicit negatives.
  • Momentum-based hard negatives: maintain a queue of recent embeddings (as in MoCo) and use the most similar items in the queue as hard negatives.

Zero-Shot Transfer and Compositionality

A critical property of cross-modal models is compositional generalization: can a model handle a novel combination of concepts (e.g., “a red cube on top of a blue sphere”) if it has seen “red cube” and “blue sphere” separately?

Standard CLIP models struggle with compositionality — they tend to match based on individual objects rather than their spatial and relational arrangement. This limitation has spawned several lines of work:

  • NegCLIP: trains with hard negative captions that swap adjective-noun bindings (“a blue cube on top of a red sphere”) to teach relational sensitivity.
  • BLIP-2: uses a lightweight Querying Transformer (Q-Former) between the frozen vision encoder and the frozen language model, enabling more structured cross-modal interaction than a single cosine similarity.
  • Structured CLIP: augments training with scene graph annotations that explicitly encode object-relation-object triples.

Applications

Cross-modal contrastive models serve as foundation models for a wide range of downstream tasks:

  • Image-text retrieval: given a query image, find matching captions (or vice versa). CLIP embeddings achieve strong performance without any task-specific fine-tuning.
  • Open-vocabulary object detection: extend detection to arbitrary class names by replacing fixed class embeddings with CLIP text embeddings (OWL-ViT, GLIP).
  • Semantic image editing: text-guided diffusion models (Stable Diffusion, DALL-E 2) use CLIP embeddings to condition image generation on text prompts.
  • Medical imaging: domain-specific CLIP variants (BioViL, MedCLIP) trained on radiology report-image pairs transfer to chest X-ray classification with few labels.
  • Robotics: CLIP embeddings serve as vision-language representations for grounding natural language instructions in robot manipulation tasks.

Summary

Cross-modal contrastive learning — exemplified by CLIP, ALIGN, SigLIP, and CoCa — trains aligned vision-language representations from large-scale paired web data using InfoNCE or sigmoid contrastive losses. The resulting embeddings support zero-shot classification, open-vocabulary detection, and semantic image retrieval without task-specific supervision. SigLIP’s sigmoid loss eliminates the distributed all-gather bottleneck of softmax contrastive learning. data2vec extends the paradigm to arbitrary modalities under a single self-supervised framework. Limitations in compositional reasoning motivate hard negative mining and structured cross-modal interaction beyond simple dot-product similarity.