Apple researchers taught an LLM to predict tokens up to 5x faster

A new research paper from Apple details a technique that speeds up large language model responses, while preserving output quality. Here are the details.

The nerdy bits

Traditionally, LLMs generate text one token at a time. This is slow because each step depends on all the previous ones to keep the output coherent and accurate.

If the model is writing a sentence like “The cat is black”, it predicts each token in sequence. After writing “The cat is”, it looks at everything…