Today marks 6 months since the launch of Artificial Analysis - so I thought I’d take the opportunity to share the story of what we’ve been building.
Artificial Analysis is an independent benchmarking, evaluation and insights provider for AI. Our benchmarks let engineers and companies make the best decisions on which technologies and providers to use, empowering them to build the next generation of AI applications.
We’ve built a benchmarking stack to measure quality and performance of AI models that tests hundreds of API endpoints every day. We publish our independent real-time analysis of language, image and voice models, and we work with the companies across the AI value chain as an independent benchmarking partner.
A handful of highlights from the last 6 months (links in comments):
‣ Being featured on the All-In Podcast, Latent Space Podcast, No Priors Podcast; being covered in VentureBeat, Gizmodo.com, SemiAnalysis and more
‣ Working with leading AI companies from chips to infrastructure to labs, including supporting recent launches from Groq and SambaNova Systems
‣ Support from industry leaders including Andrew Ng, Shawn swyx W, Chamath Palihapitiya
‣ Tens of thousands of users from using Artificial Analysis every week - from start-ups to large enterprises
‣ Appearing on a panel in San Francisco hosted by BootstrapLabs, sharing thoughts on role of benchmarking and evaluation for building trust in AI systems
‣ Hearing stories every week of how people are using Artificial Analysis to understand the AI market and build incredible applications
The story of Artificial Analysis began nearly two years ago when I embarked on building an AI legal research tool. While building analysis tools to select optimal models for different parts of my legal research algorithm, I became obsessed with the problem of understanding and comparing AI models. In early 2023, I built a couple of experimental dashboards to share some of my early work on the problem and began to develop a framework for helping engineers make trade-offs between quality, speed and price.
Late last year, I began to collaborate with my friend George Cameron (who I met interning at Google together many years ago!) to build Artificial Analysis.
I’ll be sharing more over the coming months - about what metrics matter most in AI, how developers should think about comparing AI models for scaling production applications, how we see our role in the industry evolving as the AI frontier moves forward.
As we move into the next phase, our vision is to become the definitive source for data and insights in the AI industry. To achieve this, we're growing our team and hiring now for engineering and analyst roles. If you’re excited about what we’re building here, we’d love to hear from you.
Finally - please follow Artificial Analysis on LinkedIn and Twitter to stay in the loop for our upcoming launches and analysis!