playing with local AI models and doing some research and came across a fun extrapolation:
"consumer grade graphics cards will be running Fable-equivalent models by 2029"
the argument:
1) been said that LLMs are really a lossy compression process for squeezing humanity's knowledge down into a chat box. Roughly you could say the entire internet/books/media/whatever is about ~200-300T tokens which gets "compressed" down to ~1T params of frontier models, so imagine a 400:1 or so compression in bytes. (We have a lot more video and world data, we'll discuss that later...)
2) People further quantize the models down (seemingly at decent quality) from 16-bit to 4-bit (NVFP4 ftw) and it seems pretty good. So that's more like 1,600:1 or so. Research suggests models only store ~2 bits of knowledge per parameter anyway — the weights were mostly air, which is why quantization works at all.
3) furthermore, we seem to be getting better at this compression, whether it's with MoE, pruning, etc. The Qwen 27B dense model now is equivalent to higher parameter models from a few years ago. 1,600:1 today and 1,600:1 in 5 years will have completely different results
4) I was looking to see if there's a Moore's Law thing happening here, and it's been measured: the "Densing Law" found capability-per-parameter doubles every ~3.3 months. Not sure how well it'll hold, but it says somethign like: "Every 3 months, the size of model needed to represent humanity's knowledge drops by half."
5) Not sure this holds though bc presumably, there's some kind of asymptote. Won't compress down to zero, the same way that modern image/video compression has theoretic limits too
5) Video is a zillion frames, almost zero semantic density. 99.99% of every frame is stuff a physics prior already predicts. World models will post compression ratios in the millions-to-one. But, nevertheless, there will be a ton of new facts seen simply by observing, that was never written down. But even without dealing with all this, today's text-oriented frontier LLMs are already pretty amazing
6) So the crazy idea here is that ultimately irreducible kernel representing humanity's knowledge might compress down to... tens of GB? May be small enough to fit onto a consumer grade GPU. Today, a consumer grade GPU for playing video games might have 32GB of memory on it, but the 27B parameter model that thing can run is getting smarter and smarter each year. Will the equivalent of Fable be able to run on a high-end consumer GPU in a few years?
This sounds crazy, but GPT-4 was rumored ~1.8T params, needed a rack of A100s, cost tens of millions to train. Today's open weight models with 27B parameters can do that with better data, distillation, and architecture.
That's the big question. Densing Law says "consumer GPUs might be running Fable-equivalent in 2028" and it might be possible, and even though that would be 100x? Seems nuts