SCOT: Self-Supervised Contrastive Pretraining For Zero-Shot Compositional Retrieval
Abstract
Compositional image retrieval (CIR) is a multimodal learning task where a model combines a query image with a user-provided text modification to retrieve a target image. CIR finds applications in a variety of domains including product retrieval (e-commerce) and web search. Existing methods primarily focus on fully-supervised learning, wherein models are trained on datasets of labeled triplets such as FashionIQ and CIRR. This poses two significant challenges: (i) curating such triplet datasets is labor intensive; and (ii) models lack generalization to unseen objects and domains. In this work, we propose SCOT (Self-supervised COmpositional Training), a novel zero-shot compositional pretraining strategy that combines existing large image-text pair datasets with the generative capabilities of large language models to contrastively train an embedding composition network. Specifically, we show that the text embedding from a large-scale contrastively-pretrained vision-language model can be utilized as proxy target supervision during compositional pretraining, replacing the target image embedding. In zero-shot settings, this strategy surpasses SOTA zero-shot compositional retrieval methods as well as many fully-supervised methods on standard benchmarks such as FashionIQ and CIRR.
Compositional retrieval without curated triplets
Imagine searching a product catalog with a reference image and an edit: “I want a similar dress, but yellow with fringe." The goal is to retrieve the image that satisfies both parts of the query: the visual appearance of the reference and the requested change. This is compositional image retrieval (CIR).
Most CIR systems learn from expensive, domain-specific triplets: a reference image, an edit instruction, and the exact target image. That supervision is difficult to curate and often fails to transfer to an unfamiliar domain. SCOT asks a simpler question: can we learn this behavior from the image-caption pairs that already exist at scale?
Figure 1. Traditional CIR needs annotated image-text-image triplets. SCOT learns from existing image-caption pairs and is designed to generalize to unseen domains. Adapted from the paper.
The key idea: let language create the training target
SCOT starts with an ordinary image and its caption. A frozen large language model rewrites that caption in two complementary ways:
- it produces a short modification, such as “replace the horses with motorbikes”; and
- it produces the modified caption, such as “a woman and man riding two motorbikes in the jungle.”
The model combines the reference-image representation with the modification text. Instead of requiring a matching target image, SCOT trains that composed representation to align with the modified caption in a frozen vision-language embedding space. It is also pushed away from the original caption and unrelated examples, so the requested edit has to matter.
Figure 2. Animated step-by-step flow of SCOT. The original training diagram is revealed from the input image and caption through the composition loss, then the inference diagram is revealed through gallery retrieval. The white paper canvas has been made transparent; all diagram content is from the paper.
This turns large captioned-image collections into supervision for compositional retrieval. We pretrained SCOT with roughly 290,000 image-caption pairs from COCO, Flickr30K, and the Amazon Berkeley Objects dataset, without using CIR triplets.
What does the generated supervision look like?
The generated edits are concrete and image-grounded. They change an object, color, or scene attribute while preserving the rest of the description. That gives the composition network varied examples of the kind of transformation a user may request at search time.
Figure 3. Examples of automatically generated modification text and modified captions. Green text is added or substituted; red text is removed. Adapted from the paper.
What did we find?
On FashionIQ and CIRR, two standard compositional-retrieval benchmarks, SCOT surpassed prior zero-shot methods and was competitive with many systems trained on fully annotated CIR data. The experiments also showed why the text-based target matters: using the modified caption as supervision outperformed using a retrieved image as a proxy target, while avoiding errors caused by imperfect image retrieval during training.
Why it matters
SCOT shows that a model can learn to make image search responsive to natural-language edits without collecting a new triplet dataset for every domain. By pairing existing captioned images with language-generated counterfactual descriptions, the method creates a practical path toward open-world, zero-shot compositional retrieval.
The full paper, code, and implementation details are available through the links above.