It turns out this process works on text too. Inception has trained its LLMs to take a random string of words and turn it into sentences that make sense. Diffusion LLMs still use transformers to encode meaning, but by producing whole blocks of text at once, they make transformers do more for less. “You’re still using a big transformer model, but you can predict many tokens at the same time,” says Inception’s cofounder and CEO, Stefano Ermon. “That’s why these models are so much faster and cost-efficient compared to what most other people are building today.”
The challenge was to take a technology designed for image generation and apply it to text. With images, if you need to change a blue pixel to a red one you can step through intermediate colors, says Ermon. That doesn’t work with text: “When you have ‘cat’ and ‘dog,’ there is not really something in between.”
Ermon is also a researcher at Stanford University. In 2024, he and a pair of his Stanford colleagues figured out the math to make diffusion models work with text. They trained a diffusion model that matched the performance of GPT-2—an LLM that OpenAI built in 2019—but was 10 times faster. It was enough for Ermon to spin out a company.
Today he has his sights on the big league. Inception claims its latest model, Mercury 2, performs as well as some of OpenAI’s GPT-4 models, released in 2023, but again 10 times faster. “We’re bullish about this approach because it’s the one that is going to scale up,” says Ermon.
The only things that matter are speed and cost, he adds: “Ultimately, the currency is going to be intelligence per dollar.”
Inception is not the only company betting on diffusion. Google is also experimenting with this approach and has built a prototype LLM called Diffusion Gemma. But Ermon is not worried about the competition. “I think it’s validating,” he says. “This is the future.”
04: Moving beyond words
Pathway, another startup based in Palo Alto, is perhaps the most extreme of this new bunch. It wants to free LLMs from the constraints of language.
The firm has built a type of LLM called Dragon Hatchling (named after the dragons in Terry Pratchett’s novel Color of Magic, which materialize if you think about them hard enough). Its standout result so far is a high score on a benchmark that pits LLMs against more than 250,000 very hard sudoku puzzles. Dragon Hatchling beat more than 97% of the puzzles; several leading LLMs from the top labs failed to solve any.
#startups #chasing #big #LLMs