What are the Models of Self-Supervised Learning

 

Self-Supervised Learning (SSL) is a machine learning process where the model trains itself to learn one part of the input from another part. It is also known as Unsupervised Representation Learning and is a subset of Unsupervised Learning.

The idea behind this Learning approach is to generate supervisory signals by making sense of the unlabeled data provided to it in an unsupervised fashion on the first iteration. Then, the model uses the high confidence data labels among those generated to train the model in the next iterations like any other supervised learning model via backpropagation.

SSL is particularly valuable for LLMs, computer vision, speech recognition, medical imaging, and other domains where huge amounts of unlabeled data are available but labeled data are expensive.

In this article, we are going to explore different models of Self-Supervised Learning that can applied on the data.

Autoencoder (AE)

An autoencoder (AE) is a neural network that learns to represent input data in a compact form without requiring manually labeled data. In self-supervised setups, the model is trained to solve a pretext task—like reconstructing corrupted or masked input.

The architecture behind Autoencoder (AE) consists of following three main components—

a) Encoder:

Takes the input data and reduces it into smaller form while preserving essential information. It has three layers which are:

• Input Layer: original data enters the network.
• Hidden Layers: These layers apply transformations on the input data, using weights and activation functions to capture important patterns
• Output: The encoder outputs a compressed vector known as encoding

b) Bottleneck (Latent Space):

This is the smallest layer in network that holds a compressed representation of the input data. It forces the model to focus on the most important features

c) Decoder:

It takes the compressed representation from the latent space and recreates it back into the original data form.

Generative Pre-trained Transformer (GPT) models

GPT models belong to the class of large language model (LLM) that is widely used in generative artificial intelligence chatbots. These models are based on the transformer architecture and are pre-trained on large datasets of unlabeled content, enabling them to generate novel content.

The transformer architecture is the core technology of a GPT and had solved many of the performance issues that were associated with older recurrent neural network (RNN) designs for natural language processing (NLP).

GPT models use a task called next-token prediction in Self-Supervised Learning where the model learns to predict the probability of the next token. The Learning pipeline behind Generative Pre-trained Transformer Model is as follows--

1. The process starts with the large collection of text which is unlabeled data.
2. Then, the text is converted to tokens.
3. Next, create training examples.
4. GPT converts each token into a vector called an embedding.
5. The embeddings are passed through multiple Transformer layers.
6. The attention mechanism in GPT allows the model to examine relationships between tokens.
7. The Transformer produces a representation for each position. A final linear layer converts that representation into scores for the vocabulary.
8. Then the model's prediction is compared with the actual next token. Here, the cross-entropy loss is used where the model is encouraged to increase the probability of the correct next token.
9. After calculating the loss, Backpropagation process is applied for updating the model parameters.

Bidirectional Encoder Representations from Transformers (BERT)

BERT is a language model introduced by Google where the key idea is that the model learns from unlabeled text by hiding some words and trains itself to predict the hidden words. This model was primarily designed to understand text. 

In Self-Supervised Learning, BERT is a self-supervised Transformer encoder that learns contextual language representations by predicting masked tokens from their surrounding context. The model is being trained on basis of the following tasks—

a) Masked Language Modeling (MLM):

• BERT randomly hides 15% of tokens in a sentence and learns to predict the hidden words based on their surrounding context.

b) Next Sentence Prediction (NSP):

• BERT learns to determine whether two sentences logically follow each other.
• This helps the model understand relationships between sentences, which is useful for tasks like question answering or document summarization.

Bootstrap Your Own Latent (BYOL)

BYOL is a new approach to Self-Supervised Learning that enhances image representation learning. The key idea behind BYOL is that it learns useful representations without negative examples or labeled data.

The BYOL architecture is built on basis of below Neural Networks both of which learn and interact with each other—

a) Online Network:

• Predicts the target network's representation of an image under different augmented views
• Contains the below stages—
  > Encoder
  > Projector
  > Predictor

b) Target Network:

• It updates with slow moving average of the online network
• Contains the below stages—
  > Encoder
  > Projector

Self-Distillation with No Labels (DINO)

DINO is a vision-specific self-supervised learning framework introduced by Meta AI researchers primarily built for learning powerful visual representations without labeled data. 

The key idea behind the Model is that a student network learns to match the output of a teacher network using different augmented views of the same image. It is particularly valuable in scenarios where labeled datasets are scarce or expensive to create.

DINO employs a student-teacher architecture that has the following networks—

a) Student Network:

• This network learns through backpropagation
• It has following major components:
  > Encoder: Extracts meaningful features from the image.
  > Projection Head: Transforms the encoder representation into a space where the self-supervised learning objective is applied.

b) Teacher Network:

• This network is updated using the student's parameters through an Exponential Moving Average (EMA).
• It has following components:
  > Encoder: Extracts features from its input image.
  > Projection Head: In this component, the teacher output becomes the target that the student tries to reproduce.

Conclusion

The models discussed above demonstrate the remarkable ability of Self-Supervised Learning to learn without relying heavily on human-labeled data. Whether reconstructing information, predicting tokens, or comparing different representations, they offer powerful approaches. But how far can machines go when they learn primarily from the data itself?