Foundation models in AI

In 2021, a new term emerged in the field of artificial intelligence: the "foundation model". This term, first used by researchers at Stanford's Institute for Human-Centered Artificial Intelligence, marks a fundamental shift in thinking from narrowly specialized AI models to universal, all-encompassing models. What exactly are foundation models and why are they so important?

From specialization to universality

The year 2020 brought the first visible results of this shift in thinking with the launch of the GPT-3, which is considered one of the first commercially available foundation models. According to Stanford researchers, these are unspecialized systems trained on huge volumes of unstructured data including text, images, video, and audio. These models are capable of handling a wide range of tasks.

Before 2020, neural networks (models) were trained for specific tasks such as recognizing specific objects in an image.

After about 2020, we can see a gradual shift in thinking as companies start to focus on building foundation models. Publicly known foundation models include GPT-3 and other versions from OpenAI, Llama from Meta, and Gemini from Google.

Key features of foundation models:

• The model was trained on a huge amount and variety of data.
• Multimodality - the ability to work with different types of data and generate different outputs (text, image, audio).
• Versatility - serves as a building block for diverse applications across industries.
• Adaptability - the ability to specialize in specific tasks without having to build a new model from the ground up.

Specialization of the foundation models

While these models are inherently versatile, they can be further specialized in two key ways:

• Fine-tuning - training the model with additional data to improve the accuracy of the outputs, or generating the output in a particular key.
• Prompt engineering - using precisely formulated instructions and additional information (e.g., book, article, image) to achieve desired outputs without having to re-train the model.

Advantages of foundation models

• Extraordinary performance where, with the right prompts, they can outperform specialist models in a wide range of tasks.
• Increased productivity when developing specialized applications
• Flexibility and adaptability to new tasks.

Disadvantages of foundation models

• Extreme demand on computational, energy, and financial resources in the process of training (or creating and running) the model.
• Risk of incorporating biases and inaccuracies from training data. The model may then produce inaccurate or otherwise offensive outputs.

Conclusion

Foundation models represent a significant milestone in the development of artificial intelligence. Their ability to handle a wide range of tasks with remarkable efficiency opens up new opportunities for innovation across different sectors.