Learning representations by pulling the embeddings of related pairs together and pushing unrelated ones apart, for example two augmented crops of the same photo, or an image and its caption. It needs no class labels, only a way to say which examples belong together.
Card 1 of 292. Answer: Learning representations by pulling the embeddings of related pairs together and pushing unrelated ones apart, for example two augmented crops of the same photo, or an image and its caption. It needs no class labels, only a way to say which examples belong together.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
The usual loss, InfoNCE, is a softmax cross-entropy in which the correct class is the positive example: −log(exp(sim(a, p) / τ) / Σ exp(sim(a, x) / τ)), summed over the positive and the negatives, with cosine similarity and a temperature τ. SimCLR (Chen and colleagues, 2020) used augmented views of images, with the rest of the batch as negatives; CLIP (Radford and colleagues, 2021) matched 400 million images with their captions; and the sentence embedding models used for retrieval are trained the same way.
Example: an anchor has cosine similarity 0.8 with its positive and 0.3 and 0.1 with two negatives. With τ = 0.1, the logits are 8, 3 and 1, the positive gets probability 0.99 and the loss is about 0.008. With τ = 1, the logits are 0.8, 0.3 and 0.1, the positive gets only 0.48 and the loss is 0.74, so a low temperature sharpens the contrast.
Follow-ups: it needs many negatives, hence large batches (SimCLR used 4,096) or a queue of past embeddings as in MoCo; without negatives every embedding could collapse to the same point, which BYOL and SimSiam avoid by other means; and in retrieval, hard negatives that look relevant but aren't matter most.
