As the AI race between companies, nations, and ideologies continues apace, Nvidia has released a paper describing TiDAR, a decoding method that merges two historically separate approaches to accelerating language model inference. Language models produce text one token at a time, where a token is a small chunk of text, such as a word fragment or punctuation mark.
Each token normally requires a full forward pass through the model, and that cost dominates the speed and expense of running today’s AI…