These startups are chasing the next big thing in LLMs

TL;DR

Transformers, the core technology behind current LLMs, are computationally expensive and have limitations in processing long sequences of data and maintaining large context windows.
Startups are developing new methods to address these limitations, including sparse attention, power retention, liquid neural networks, diffusion models for text generation, and state space models.
These innovations aim to create faster, more efficient, and potentially more capable LLMs, challenging the current technological paradigm.
Companies like Subquadratic, Manifest AI, Liquid AI, Inception, and Pathway are pioneering these new approaches.
The ultimate goal is to create AI systems that are not only more cost-effective but also capable of more advanced and novel forms of reasoning.

Continuer la lecture de l'article original