tech
Google's Latest DiffusionGemma Open AI Model Comes with a 4x Speed Boost
Diffusion AI is Most Common in Image Generation, But It Can Make Text Outputs Much Faster.
TL;DR
- Google DeepMind's DiffusionGemma generates text in parallel blocks, unlike traditional left-to-right autoregressive models.
- This parallel generation method makes DiffusionGemma faster and more efficient on local hardware, including gaming GPUs.
- The model is a 26 billion parameter Mixture of Experts (MoE) model, with 3.8 billion activated during inference, fitting within 18GB RAM.
- DiffusionGemma can produce over 1,000 tokens per second on an Nvidia H100, approximately four times faster than similarly sized autoregressive Gemma models.
- This approach shifts the bottleneck from memory bandwidth to compute, enabling faster processing for tasks like in-line editing and molecular sequencing.
- Drawbacks include a higher error rate in discrete text generation and inefficiency for generating short text outputs.
- DiffusionGemma is an experimental model released under the Apache 2.0 license and is available for download.
- The model is optimized for various setups, including high-end RTX GPUs and enterprise systems like the H100 or DGX Spark platform.