Cerebras has set a new record for AI inference speed, serving Llama 3.1 8B at 1,850 output tokens/s and 70B at 446 output tokens/s.
Cerebras Systems has just launched their API inference offering, powered by their custom wafer-scale AI accelerator chips.
Cerebras Inference is achieving the fastest speeds we have ever benchmarked on Artificial Analysis for Llama 3.1 8B and 70B. Pricing is also competitive at $0.1 per 1M tokens for Llama 3.1 8B and $0.6 per 1M tokens for Llama 3.1 70B. Cerebras is currently serving both models with an 8K context window (compared to the the Llama 3.1 series’ native 128k context).
Cerebras’ Llama 3.1 8B offering is nearly 10x faster than the speeds offered by OpenAI, Google and Anthropic for their current small models - GPT-4o mini, Gemini 1.5 Flash and Claude 3 Haiku.
Cerebras Inference is powered by the Cerebras WSE-3, a custom 5nm AI chip built on a unique wafer-scale design. A single WSE-3 chip is over 50x larger in total area than a Nvidia H100 and hosts 900,000 cores with 44GB of on-chip memory (SRAM).
Faster AI inference enables a new generation of AI applications, from sophisticated agentic workflows to instant search experiences.
See below for further charts and links to our full benchmark results.
Groq Thanks for Sharing! 😃