Two quick definitions first
When you ask an AI a question, it writes the answer piece by piece. That process is called inference, and the pieces are called tokens. A token is roughly three-quarters of a word. So a model running at “200 tokens per second” is producing about 150 words every second.
The next race
For years, AI companies competed on one thing: making models smarter. bigger, more training data, the ability to read more at once (bigger context window sizes).
That competition is still going. But a second one has started slowly, and it’s about speed. Faster inference, answers & response times
What the big players are doing
- Google built Gemini 3 Flash, which runs at around 218 tokens per second. A comparable OpenAI model does about 125.
- Anthropic Claude Opus 4.8 has a “fast mode” that runs the identical model about 2.5x faster. Same quality, roughly double the price, arrives sooner.
- OpenAI is chasing the same thing. Bigger models are normally slower, so to keep GPT-5.5 as fast as the smaller model it replaced, the company rewrote its serving software and moved onto NVIDIA’s newest GB200 and GB300 chips.
- Cerebras, a chip company, has pushed top models to nearly 980 tokens per second. That’s about five times faster than the leading mainstream services.
Think about mobile networks
This has happened before, in a different industry. Mobile networks went from 1G to 5G. Each step brought more speed and less delay. The result wasn’t just that old apps ran faster. We got entirely new things: video streaming, ride hailing, real-time video calls.
Three metrics you’ll start hearing
| Metric | What It Measures | Why It Matters |
|---|---|---|
| TTFT (Time to First Token) | How long you wait before the first word appears. | Determines how responsive the AI feels. A high TTFT creates the “Is this thing frozen?” moment. |
| Tok/s (Tokens per Second) | How quickly the model generates the rest of the response after it starts. | Higher throughput means answers finish sooner and conversations feel more fluid. |
| ITL (Inter-Token Latency) | The delay between consecutive generated tokens (words or subwords). | Low, consistent ITL produces smooth streaming. High or inconsistent ITL causes responses to appear in jerky bursts instead of flowing naturally. |
Why it matters
Waiting compounds. A chatbot answers once, but an AI agent working through a task on its own might call the model 30 or 40 times in a row, each step waiting on the last, so saving a second per step turns a 90-second job into a 30-second one. Beyond that, faster inference is poised to unlock the next generation of AI experiences. With ultra-fast inference, LLMs start to feel less like software you wait for and more like something you interact with in real time: better multi-agent systems, voice agents you can interrupt mid-sentence, emergency response tools that read, analyse and act in seconds, and a whole new kind of app experience.
Intelligence Is Becoming a Commodity
Every major frontier model is remarkably capable. They’re all getting smarter. But the improvements are becoming increasingly incremental.
Inference performance, on the other hand, still has enormous room for improvement. Some specialized inference platforms already deliver throughput several times higher than traditional hosted APIs.
That suggests we’re only beginning to explore what’s possible. Eventually, raw intelligence won’t be the deciding factor for most applications. Responsiveness will.