Generative AI uses an approach called machine learning, what that means is these systems all learn by example. And to do that, they need a lot of examples. So the companies developing any of these generative AIs have to begin by gathering massive, massive amounts of data to analyze and learn from, whether that's text or images or code.

A text‑based generative AI is built by analyzing millions of text sources like web pages, books, articles, scientific papers; whereas an image‑based generative AI will have been trained on countless images, photographs, illustrations diagrams, and they need staggering amounts of data because they're trying to identify patterns and similarities in the data. And the combination of all the data that they're trained on, together with your suggestions or your prompts, allows them to generate that brand new content.

People often wonder what do you actually call this? I mean is it a program? Is it an application? what is it? Well, we call it a model. The end result of this training process, which is very expensive and very time‑consuming, is a model. We say that we have trained the AI model on all of this data.

Model: An AI system that uses data to find patterns and make predictions.

There's even more specific terms. When a generative AI is text‑based, meaning it's been trained on incredible amounts of text and language, we call it a large language model, or LLM.

Large Language Model (LLM): An AI that processes and generates human-like text, learning from vast textual datasets to understand and mimic language.

But if the model was trained on some other kind of data like images or video, we're not going to call it a large language model because it isn't dealing with language. The term you'll often hear is foundation model.

Foundation Model: A versatile AI model trained on extensive, diverse data, adaptable for broad applications and tasks.

Training a generative AI model is very time‑consuming and extremely expensive. The large commercial generative AIs, the ones used by ChatGPT or Google Bard, they are estimated to cost tens of millions of dollars in computer costs alone just to train the model. They need these massive server farms and tons of equipment to do it, and it's one of the reasons why there is a cut‑off date with what ChatGPT knows about because that LLM took so long to train. It was trained up to a specific date. You can't just do it again the next day because it costs incredible amounts of money to train them.

It's a common misconception to think that because these generative AIs are trained on millions of existing examples that they store all of that data they're trained on, but they don't. They try and recognize patterns in the original data, but they do not store all the original document. So a generative AI model is not like a database where you can just look something up.

Here's what I mean. If I was a generative AI model, I've analyzed millions of documents including multiple documents that contain the phrase "it was a dark and stormy night". Now I could recognize this pattern, recognize this occurrence of words one after the other. And after being trained, if you then ask the model, well, what comes after the words "it was a dark and stormy"?, the model will say, well, "night" probably.

I mean statistically, if somebody writes "it was a dark and stormy", the next word is probably going to be "night". However, if you now ask the model, "okay, now tell me every single document that contained that phrase and the rest of the contents of those documents", it doesn't know because it didn't actually store all the documents it analyzed. It stored the results of the analysis. So while it's really good at recognizing patterns and making new predictions, it is not good at just looking something up.

While it might be an oversimplification to say that generative AI is like a big autocomplete system, there is an element of truth to that. Fundamentally, it is trying to figure out if you wrote these words or these sentences, what is the statistically most likely next word or sentence or paragraph? But also, because the model is trained on massive amounts of data, everything from scientific papers to novels, song lyrics, equipment manuals, the model doesn't actually care whether it's looking at the text for a washing machine manual or looking at the text for a fantasy novel. It's just text. And an LLM does not make a distinction between fiction and non fiction. It doesn't understand true and false. It is just trying to recognize patterns in the data.

Let's take another phrase like "that was a massive waste of", and what comes next?

Massive waste of time? Massive waste of money? Massive waste of time and money? Massive waste of my time" your time? the company's time? All of these are possible, and they would all work as the next word or phrase after that beginning. But if you had previous context, let's say you also knew all the sentences prior to this one, it allows the predictions to be better and more accurate. And it's why a lot of LLMs will talk about having a larger and larger context length or context window, meaning what is the overall amount of things that they remember? What is in their working memory? What did they understand about this particular interaction? But here's the thing. This idea that there are often multiple possible answers is why generative AI can give you results where you understand why you got that result, but it's completely wrong.