16000 Babies Trying Their Best

I wondered for a while why we use VRAM for AI as opposed to regular RAM, and what the difference was. VRAM to me is always associated with, well, video quality. When you're playing a game, you want a better video card to show off those beautiful graphics. But you also want the RAM in there so that it's handling some of the bigger tasks in the game. The bigger the RAM, the more complex the task. So what do graphics have to do with Large Language Models? And why doesn't regular RAM cut it? Well, as it turns out, while regular RAM is your hyper intelligent, complex task and calculation runner, it's limited by scope. VRAM, on the other hand? It is Legion. It is plentiful. It is mentally stunted. Let's talk about that.

The Babies Take Over

While regular RAM acts more like a few seasoned professors, able to calculate vast information and be exceptionally accurate, they are restricted by the amount of stuff they can actually do in a given moment. It is only a handful of professors, with two hands each, and they can only work so fast at the equation. However, if you get 16,000 babies in a large enough room (the VRAM), give them each a tiny crayon, and tell them "solve for this complex social problem," they'll bang it out quickly. It may not always be right, but it'll work for what you're wanting.

This is why VRAM for local compute power is so important. You throw in a large number of computations, and if the babies have room to move around in your VRAM storage, they can get the problem done without getting stalled, tripping over each other, and generally make lightning fast calculations. But if the space is too small, you have lower end VRAM, you don't have the space? Then they'll try to run down the hall, and ask the professors for help, which creates bottlenecks, slows down the system, and generally makes the whole thing unstable.

That isn't to say that regular RAM isn't important: sometimes the babies just need some professors to help them out. The problem is just that you need a lot of babies in order to process at fast speeds what the hell is actually going on in that moment.

Tiny Babies Need Tiny Crayons

The next part of this boils down to how the VRAM Babies handle large bits of information. If you give them massive quantities of data all at once, it's basically handing them a giant, comically sized crayon. They'll pick it up, sure, but then they'll not be able to carry it, drop it, and start crying. And nobody, systems included, wants a crying baby.

So, when we "tokenize" the data, we're basically chopping the crayons down into tiny, baby-hand sized crayons, that they can't choke on, but they can use to write on the giant wall that is your response. These chunks are usually broken down into 4 or 8 bit "crayons" that let the babies handle the heavy lifting with ease.

The tiny crayons may sometimes lose a bit of nuance, because they're just getting small bits at a time, but this is all so that the babies can fully write on the wall, with the ease and access of baby-hand crayon technology.

As our tech gets better, the babies improve their toolkits (think diaper bags, self serve bottles, etc.), but they're still babies with tiny crayons. The professors can step in from time to time, but the tiny crayon babies are literally the best folks for the job right now.

Your Own Personal Baby Playground

This is the fun part: where you learn to build your own room full of babies. By bringing your LLM to your computer (hopefully with enough VRAM to let the babies really play), you're basically building a playground for these babies to roam free. They don't toddle into sharing data, because your playground is walled off from the general internet, except if you need it to do quick searches. And even then, a couple hundred babies in a trench coat, asking internet strangers for help is going to be more helpful for you than the stranger.

When choosing a model (the number of babies) you want to pick one that fits the size of your playground. That's why finding the right size of model is as important as getting the right sized video cards. Thousands of babies need lots of room to play, even if the professors sometimes have to play nanny.