Large language model (LLM) inference often struggles with speed because models typically generate text one token at a time. Speculative decoding attempts to fix this by using a smaller "drafter" model to guess several tokens at once, which the larger "target" model then verifies in a single step.

Traditional methods like EAGLE are autoregressive, meaning the drafter still works sequentially, creating a hidden bottleneck as sequences grow longer. P-EAGLE (Parallel-EAGLE) shatters this performance ceiling by producing all draft tokens in a single forward pass, significantly reducing the time spent guessing.

To make this work, the researchers introduced a novel architecture that uses "mask" tokens as placeholders for future predictions. These placeholders are processed together through the model layers, allowing the system to look ahead without waiting for previous tokens to be finished.

This breakthrough is now live in the vLLM serving engine, a popular open-source tool for running AI models. Early tests on NVIDIA's powerful B200 hardware show substantial throughput gains, particularly for complex tasks like coding and multi-turn conversations where long outputs are common.

Users can leverage pre-trained P-EAGLE heads for models like GPT-OSS and Qwen3-Coder immediately. This advancement signals a shift toward more efficient, parallelized inference techniques that could make real-time AI interactions much smoother and less expensive for developers to operate.

Large language model (LLM) inference often struggles with speed because models typically generate text one token at a time. Speculative decoding attempts to fix this by using a smaller "drafter" model to guess several tokens at once, which the larger "target" model then verifies in a single step.

Traditional methods like EAGLE are autoregressive, meaning the drafter still works sequentially, creating a hidden bottleneck as sequences grow longer. P-EAGLE (Parallel-EAGLE) shatters this performance ceiling by producing all draft tokens in a single forward pass, significantly reducing the time spent guessing.

To make this work, the researchers introduced a novel architecture that uses "mask" tokens as placeholders for future predictions. These placeholders are processed together through the model layers, allowing the system to look ahead without waiting for previous tokens to be finished.

This breakthrough is now live in the vLLM serving engine, a popular open-source tool for running AI models. Early tests on NVIDIA's powerful B200 hardware show substantial throughput gains, particularly for complex tasks like coding and multi-turn conversations where long outputs are common.

Users can leverage pre-trained P-EAGLE heads for models like GPT-OSS and Qwen3-Coder immediately. This advancement signals a shift toward more efficient, parallelized inference techniques that could make real-time AI interactions much smoother and less expensive for developers to operate.