Generative Artificial Intelligence (AI) is a field that has witnessed remarkable progress in recent years, transforming the way we perceive and interact with technology. It encompasses a diverse range of techniques and algorithms that enable machines to generate new content, including images, music, text, and even human-like conversations. The evolution of generative AI has been driven by a combination of innovative research, advancements in computing power, and access to vast amounts of data. It has a power to revolutionise industries like music, video, content creation, entertainment, finance, agriculture, health care and many more.  Examples of Generative AI models include ChatGPT, which is a language model that can understand and respond to human language, and DALLE-2, which generates high-quality images based on textual descriptions. Let’s delve into the fascinating history of generative AI and explore its major milestones.

Early Beginnings:

The roots of generative AI can be traced back to the 1950s when computer scientists and mathematicians began to explore the concept of machine learning and artificial intelligence. Pioneers such as Alan Turing and John McCarthy laid the foundation for generative AI by proposing early models of computation and the idea of machines that could mimic human intelligence.

Generative AI models can be classified as Unimodal or Multimodal.

1. Unimodal models generate output in the same format as their input,
2. multimodal models can take input from different sources and generate output in various forms

source : https://arxiv.org/pdf/2303.04226.pdf

Generative models have a long history in AI, starting with Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs) in the 1950s. However, significant advancements in generative models came with the introduction of deep learning techniques.

In the field of Natural Language Processing (NLP), generative models have evolved from N-gram language modeling to recurrent neural networks (RNNs) such as LSTM and GRU, which can generate longer sentences and capture longer dependencies in language.

For computer vision tasks, traditional methods relied on texture synthesis and mapping techniques, but the introduction of Generative Adversarial Networks (GANs) in 2014 revolutionized image generation. Variational Autoencoders (VAEs) and diffusion generative models have also been developed to enhance control and quality in image generation.

The transformer architecture, originally introduced for NLP, has become a fundamental building block for generative models. Transformers like BERT and GPT are large language models used in NLP, while Vision Transformers and Swin Transformers combine transformer architecture with visual components for image-based tasks. Transformers have also enabled the fusion of models from different fields for multimodal tasks, such as CLIP, which combines vision and language to generate text and image data.

Timelines

source : https://arxiv.org/pdf/2303.04226.pdf

Here is a point-by-point summary of Generative AI from GAN to ChatGPT

1. N-Gram: Developed in the 1960s & 1970s, an N-gram model is a statistical language model used in NLP tasks to estimate word sequence probabilities based on frequency calculations.

2. Long Short-Term Memory (LSTM): Introduced in 1997, LSTM is a type of recurrent neural network (RNN) designed to capture long-term dependencies in sequence prediction tasks by processing entire sequences of data.

3. Variational AutoEncoders (VAEs): Released in 2013, VAEs are generative models that compress data into a smaller representation and generate new samples similar to the original data.

4. Gated Recurrent Unit (GRU): Developed in 2014 as a simpler alternative to LSTM, GRU is a variation of RNN that processes sequential data with the use of gating mechanisms to selectively update the hidden state.

5. Show-Tell: Released in 2014, Show-Tell is a deep learning-based model that combines computer vision and machine translation techniques to generate human-like descriptions of images.

6. Generative Adversarial Network (GAN): Introduced in 2014, GANs consist of a generator and a discriminator and can generate new data points resembling the training data.

7. StackGAN: Released in 2016, StackGAN is a neural network that generates realistic images based on text descriptions by stacking two GANs together.

8. StyleNet: Introduced in 2017, StyleNet is a framework that generates attractive captions for images and videos with different styles by learning the relationship between visual content and natural language captions.

9. Vector Quantised-Variational AutoEncoder (VQ-VAE): Developed in 2017, VQ-VAE is a generative model that learns useful representations by outputting discrete codes and learning the prior.

10. Transformers: Released in 2017, transformers are neural networks that analyze relationships between words to understand the context of sequential data, such as sentences.

11. BiGAN: Introduced in 2017, BiGAN is a bidirectional generative adversarial network that can create realistic data by learning from examples and mapping the data back to its original representation.

12. RevNet: Developed in 2018, RevNet is a deep learning architecture that can learn representations without discarding unimportant information by using homeomorphic layers and explicit inverse functions.

13. StyleGAN: Released in 2018, StyleGAN is a GAN that generates high-quality images by progressively adding details based on style vectors and noise inputs.

14. ELMo: Introduced in 2018, ELMo creates word vectors based on entire sentences to capture the context of words in different contexts.

15. BERT: Released in 2018, BERT is a language representation model pre-trained on large amounts of text, enabling transfer learning for various NLP tasks.

16. GPT-2: Developed in 2019, GPT-2 is a transformer-based language model with 1.5 billion parameters capable of generating high-quality synthetic text samples.

17. Context-Aware Visual Policy (CAVP): Introduced in 2019, CAVP is a network designed for fine-grained image-to-language generation, considering previous visual attention as context.

18. Dynamic Memory Generative Adversarial Network (DM-GAN): Released in 2019, DM-GAN generates high-quality images from text descriptions and includes a dynamic memory module for refining image contents.

19. BigBiGAN: Developed in 2019, BigBiGAN is an extension of GAN architecture focusing on image generation and representation learning.

20. MoCo: Introduced in 2019, MoCo is an unsupervised learning method.

Some of the import breakthroughs in recent year

1. VisualBERT (2019, Vision Language): A framework that combines language understanding and image comprehension using self-attention. It performs well in tasks like image question-answering and image description.

2. ViLBERT (2019, Vision Language): A model that integrates visual and textual information through co-attentional transformer layers. It can answer questions about images, understand common sense, find specific objects, and describe images in text.

3. UNITER (2019, Vision Language): Trained on large datasets of images and text, UNITER achieves state-of-the-art results in tasks like image question-answering, object finding, and common sense understanding.

4. BART (2019, NLP): BART is a sequence-to-sequence pre-training model based on the Transformer architecture. It performs well in text generation and comprehension tasks and excels in summarization, question-answering, and dialogue tasks.

5. GPT-3 (2020, NLP): Developed by OpenAI, GPT-3 is a massive language model with over 175 billion parameters. It generates highly sophisticated text with minimal input and represents a significant improvement over previous language models.

6. T5 (2020, NLP): T5 is a text-to-text Transformer architecture used for various NLP tasks such as question answering, translation, and classification. It offers a unified approach, using the same model and hyperparameters for different tasks.

7. DDPM (2020, CV): DDPM is a diffusion probabilistic model used for generating high-quality images. It leverages a method called lossy decompression.

8. ViT (2021, CV): ViT is a visual model based on transformers, originally designed for text tasks. It processes images by dividing them into patches and achieves impressive results, surpassing traditional Convolutional Neural Networks (CNNs).

9. CLIP (2021, Vision Language): CLIP is a neural network that uses natural language supervision to efficiently learn visual concepts. It can be applied to various visual classification benchmarks and exhibits zero-shot capabilities.

10. ALBEF (2021, Vision Language): ALBEF is a vision and language representation learning approach that aligns and fuses image and text representations through cross-modal attention. It achieves state-of-the-art performance in vision-language tasks.

11. VQ-GAN (2021, Vision Language): VQ-GAN is an extension of VQ-VAE that generates high-resolution images using a patch-wise approach and perpetual loss for increased perceptual quality.

12. DALL-E (2021, Vision Language): DALL-E is a machine learning model that generates images from textual descriptions. It demonstrates impressive abilities in creating diverse and realistic images based on text inputs.

13. BLIP (2022, Vision Language): BLIP is a Vision-Language Pre-training framework that achieves state-of-the-art results on vision-language tasks. It effectively utilizes noisy web data by bootstrapping captions.

14. DALL-E 2 (2022, Vision Language): DALL-E 2 is an advanced AI model that generates high-resolution images from textual descriptions using a GPT-3 transformer model.

15. OPT (2022, NLP): OPT is a suite of decoder-only pre-trained transformers that range from 125M to 175B parameters. It aims to make large language models more accessible to researchers.

16. Sparrow (2022, NLP): Sparrow is a dialogue agent developed by DeepMind that engages in conversations, provides answers, and searches the internet for supporting evidence to enhance responses.

17. ChatGPT (2022, NLP): ChatGPT is an open-source chatbot powered by

GPT-3. It is trained on various topics, capable of answering questions, providing information, and generating creative content in different conversational styles.

18. BLIP2 (2023, Vision Language): BLIP2 is a pre-training strategy that combines pre-trained image encoders and large language models for efficient vision-language pre-training.

19. GPT-4 (2023, NLP): OpenAI’s latest system, GPT-4, is more useful and safer than its predecessors. It has an enhanced knowledge base, problem-solving abilities, and creativity, making it collaborative in generating and editing creative and technical writing tasks.

These models span across vision-language tasks, natural language processing, computer vision, and dialogue systems, each with its unique contributions and advancements in their respective categories.

Future Prospects:

Looking ahead, the future of generative AI appears promising. Researchers are exploring new techniques, such as reinforcement learning and unsupervised learning, to further enhance the capabilities of generative models. Advancements in hardware, such as specialized AI accelerators and distributed computing, will continue to push the boundaries of generative AI and make it more accessible to a broader range of applications and industries.

In conclusion, the history of generative AI showcases a remarkable journey of innovation and progress.