By Maia Salti
(Image generated by Google Gemini - Nano Banana Pro)
Since I have rebranded my blog from "Data by Maia" to "Maia Talks About AI," I thought it was appropriate that I begin with the most widely known type of AI: LLMs.
ChatGPT, Claude, Gemini, Grok, Llama, DeepSeek: these are all LLMs (Large Language Models).But how many of the 250 million daily active ChatGPT users know how an LLM actually works?
Well, I spent a while using an LLM to teach me how LLMs work (how meta) and here's my attempt to explain it as simply as I can. I also added in optional dropdowns for anyone who wants to read more about the math for each section.
Just a disclaimer: I skipped over a bunch of steps for the sake of not overcomplicating the explanations, but the main concepts are here.
#1 Tokenization
Firstly, an LLM doesn't look at words and sentences the same way that we do.
Before the model does anything, the text gets chopped up into chunks (not necessarily words) called tokens.
Each token gets converted to an ID (a unique number), meaning the sentence gets turned into a list of integers.
To create these tokens, the model starts by scanning a large pile of text, and learning which chunks show up often.
The standard method, called Byte Pair Encoding, starts with individual characters (‘a’, ‘w’, ‘r’), then repeatedly finds the most frequent pair of characters and turns that group into a single new token. Pairs become groups of three/ four/ five and some stay in pairs, It does this tens of thousands of times.
Frequent words keep getting promoted and eventually get their own single-token slot.
For example, start with 'h'. As it scans through text, it notices 'he' comes up very frequently, so 'he' becomes a single token. Then as it keeps scanning, it finds that 'the' comes up frequently, so that becomes a single token.
Whereas if you have a token like 'inter' and 'international' doesn't come up often enough in the text, 'inter' will stay as a single token and 'national' may become a separate token.
#2 Embeddings
For simplicity of explaining embeddings, I’ll refer to ‘tokens’ as ‘words’ instead as it's easier to understand that way.
Now each word is given a set of coordinates that place it on a map (picture it kind of like a web of words on a graph). Words that have similar meaning, like King and Queen, would sit close together on this map.
However, your typical graph (horizontal x-value, vertical y-value) isn't sufficient for mapping words. There are many factors defining the "meaning" of a word. It's like trying to fit all the words in the dictionary into two different categories (e.g. positive and negative): it doesn't really describe the words very well to categorize them that way.
Real embedding coordinates can be hundreds or thousands of dimensions to account for the complexity of the English vocabulary and how it's used. (It really gives you a better appreciation of what our brains are capable of.) The higher dimensions allow for separation of the different semantic meanings and contexts.
That is essentially embedding. It gives every token a list of numbers (a vector) that actually gives it meaning. The model learns the position of each word during training.
#3 Attention
Now each word has a meaning assigned to a vector. But the issue is, a word can have different meanings depending on context. For example, "bank" by a river and "bank" with money have the same vector, but have different meanings.
Attention fixes this by letting every word look around at all the other words in the sentence and decide which ones are relevant to it, and pulls in a blend of their information to update its own vector.
Each word is going around talking to other words in the sentence to figure out its own context.
#4 Training
Training, which is essentially giving the model the "rules" so that it can spit out the right text, is based on this premise: predict the next word.
We grab a section of the text, hide the next word, and then let the model guess. It will output a probability for every possible next token. It will then guess the word it has calculated with the highest probability and then compare it to the word that actually came next.
It will then be rewarded if it's right and penalized if it's wrong. If it was confidently wrong it will be penalized more so than if it was only slightly confident (lower probability). This is called the loss: how wrong the model was, or, mathematically, the distance between the true data points and the model prediction
We then adjust every parameter, like adjusting a straight line on a graph so that it goes through as many points as possible, to minimize this loss using gradient descent (a topic I'll cover in another blog post).
Click around on this graph to explore how loss really works.
In this example, the blue line is the model. If the data points weren't there, you could use the line to predict where the data could be. That's the premise of training.
#5 Final Prediction
The vector of the last word in the sentence is multiplied by a final weighted matrix, which then outputs a probability for each word in the vocabulary. That probability represents the chance of that word being the next correct word.
E.g. "The Sky Is …" → Blue = 0.86, Falling = 0.08, Red = 0.03, Not = 0.02, Here = 0.01
The model picks out of the most likely words (which is why one prompt can generate different answers, it's not necessarily the word with the highest probability that's selected), and that's its answer.
In short
An LLM is a massive system of numbers that has learned over billions of examples, to predict the next word in a sequence. There is a lot of adjusting on top of that base to make it safe, friendly, and ChatGPT-sounding, but you understand the foundation now. I hope this was helpful!
If you're experienced with LLMs and felt like something here wasn't explained well or that I got something wrong, please email me at maia.salti@gmail.com. I'd love to learn as much as I can.
All diagrams were originally handwritten digitally, then converted to text with Google Gemini Nano Banana Pro.