Maia Talks About AIMaia Talks About AI

Attention Is Still All You Need

2026-07-29

By Maia Salti

Attention is still all you need Generated by Gemini nano banana

You know it's a great day for AI when Peter Diamandis uploads an emergency episode on his Moonshots podcast. If you haven't heard of it, go listen to a few of the most recent episodes. Those guys really fire me up and keep me thinking optimistically about where AI is heading.

Diamandis pulled together his full panel for this episode: Salim Ismail, Dave Blundin, Emad Mostaque, and Alexander Wissner-Gross (sidenote: my father also highly recommends Alexander's Substack, The Innermost Loop. He says it's the one place to know everything about AI developments). They half-jokingly called the model release an AI Sputnik moment, although at some point they argued it was even larger than that.

A couple of articles ago, I wrote about DeepSeek and how they worked within the constraints of U.S. export controls and still managed to release a powerful model that was competitive with the U.S. frontier models of that time. This is that, but maybe 10x.

Meet Moonshot AI's Kimi K3: an open-weight, 2.8-trillion-parameter model with a 1-million-token context window. On Artificial Analysis's private evaluations it landed an Elo of 1547, behind only Anthropic's Claude Fable 5. It beat Claude Opus 4.8 and GPT-5.5 on most tasks, and it beat Fable 5 outright on the Frontend Code Arena benchmark. It costs $3 per million input tokens and $15 per million output tokens, against Fable 5's roughly $50 for the same output. Visit my site waitwhichmodel.fyi for more info.

In the podcast episode, Wissner-Gross was talking about the architecture and noted that there's really not a massive LLM structural change here. It's still a transformer. Then he goes, "I guess attention is still all you need" (hence the article name). He then asks another question that I want to explore further: "What are the American frontier labs spending their money on? If you can just use a transformer to get this close, it's already on the cost frontier."

Summary of How

Here are some brief bullet points on the context and what you need to know:

Background

  • Moonshot trained its earlier K2 models on NVIDIA's H800, the deliberately slowed-down chip NVIDIA built to comply with the 2023 export restrictions.
  • K3's own documentation is vague, citing H200s and "a GPGPU from an alternative vendor," potentially Huawei's Ascend line. Whatever it is, it isn't what OpenAI, Anthropic, and Google get to train on.

K3 architecture changes

  • Kimi Delta Attention: swaps the usual all-tokens-compare-to-all-tokens mechanism for a cheap linear recurrence most of the time.
  • Muon (MomentUm Orthogonalized by Newton-Schulz): instead of updating each weight direction independently (the way Adam does, which amplifies the direction with the loudest gradient), it forces the whole update into an orthogonal matrix, so every direction gets an equal say regardless of who was shouting loudest. Cheap to compute, adds under 1% overhead, and Moonshot reports a 35% training speedup over AdamW.

Frontier Labs

So is this the commoditization of LLMs and the downfall of frontier labs? I prompted Fable 5 to ask me a bunch of questions and bullet-point my general thoughts, so here are my predictions (and the only AI-generated section of this article):

Where value goes

  • Model margins fall to near-zero as inference gets cheap. Profit moves to proprietary data, real-world execution, and cutting through bureaucratic slowness.
  • The model layer levels out, but who wins is still open. New players win if speed and execution matter most. Giants win if distribution, lock-in, and capital matter most — and history (cloud, mobile) says giants usually absorb the shift. Training data is the swing factor: if it stays scarce, the giants win; if synthetic data fixes it, the field stays open.
  • Frontier labs pivot to agents and orchestration — but only if that layer doesn't get commoditized too. Brand trust might be the moat. Whether it holds when the model underneath doesn't is the open question.
  • A bubble likely pops, since valuations assumed intelligence would stay scarce. We just don't know when.

The next edge

  • What stays scarce: data and real-world execution now, electricity later. Speed of iteration beats owning a model.
  • Efficiency (Kimi over brute scale) favors small, disciplined players. Well-funded labs lose if they just keep throwing money and scale at problems.
  • Open-weight flooding is strategic and cuts both ways. It hurts the U.S. frontier lead, but may help U.S. app-layer players — where the real value is. Depends on whether the U.S. edge is in models or in diffusion.

Dynamics & risk

  • Businesses live and die fast and cheap. Lower cost of failure, more freedom to build.
  • The core danger is spread: dangerous capability, up to bioweapons-class, reaching everyone. This assumes attack beats defense — which isn't settled. And defense only works if institutions take the threat seriously early.
  • Power (state vs. lab vs. individual) is a separate question from money — and it's where the real risk sits.
  • The window to slow this is likely closed. A "UN for AI" is the wish, but regulation and geopolitics are the only real brakes.

In general, I think that any sort of prediction is rather useless in the case of AI development. Human brains are not trained to predict exponential growth very well, and I don't think many people are really able to comprehend the changes that will come in the next 5–10 years.

My only opinion of Kimi K3 is positive, because competition is always good and it lights a flame underneath a lot of bottoms, prompting people to develop and grow AI with a larger sense of urgency.

This relates to the open letter Jensen Huang promoted on July 24 in his first-ever post on X. The letter, "Open Weights and American AI Leadership," argues that downloadable model weights should be treated as strategic infrastructure.

Jensen Huang's first post on X, sharing the Open Weights and American AI Leadership letter Jensen Huang's first post on X, July 24, 2026

The letter has four main points:

  1. Open weights broaden access by letting any organization use AI without the cost of training frontier models.
  2. They strengthen competition by widening the field of who can build and optimize (similar to the benefits of open-source software back in the 1980s).
  3. They support safety by allowing everyone to test for vulnerabilities.
  4. They reduce the risk of concentrated AI power.

Twenty-five companies signed at launch, including NVIDIA, Microsoft, Meta, IBM, Dell, Palantir, and Y Combinator. OpenAI, Anthropic, Google, and xAI were all absent, and that absence was most of the coverage. It didn't last long: the list doubled to 50 within a day, with OpenAI and Google both joining. Amazon and Anthropic did not.

Well, what an exciting time to be alive.

Sources

Get new posts in your inbox
I'll send you a short email when a new post is up. No spam, just the link.