Artificial Analysis reposted this
Today marks 6 months since the launch of Artificial Analysis - so I thought I’d take the opportunity to share the story of what we’ve been building. Artificial Analysis is an independent benchmarking, evaluation and insights provider for AI. Our benchmarks let engineers and companies make the best decisions on which technologies and providers to use, empowering them to build the next generation of AI applications. We’ve built a benchmarking stack to measure quality and performance of AI models that tests hundreds of API endpoints every day. We publish our independent real-time analysis of language, image and voice models, and we work with the companies across the AI value chain as an independent benchmarking partner. A handful of highlights from the last 6 months (links in comments): ‣ Being featured on the All-In Podcast, Latent Space Podcast, No Priors Podcast; being covered in VentureBeat, Gizmodo.com, SemiAnalysis and more ‣ Working with leading AI companies from chips to infrastructure to labs, including supporting recent launches from Groq and SambaNova Systems ‣ Support from industry leaders including Andrew Ng, Shawn swyx W, Chamath Palihapitiya ‣ Tens of thousands of users from using Artificial Analysis every week - from start-ups to large enterprises ‣ Appearing on a panel in San Francisco hosted by BootstrapLabs, sharing thoughts on role of benchmarking and evaluation for building trust in AI systems ‣ Hearing stories every week of how people are using Artificial Analysis to understand the AI market and build incredible applications The story of Artificial Analysis began nearly two years ago when I embarked on building an AI legal research tool. While building analysis tools to select optimal models for different parts of my legal research algorithm, I became obsessed with the problem of understanding and comparing AI models. In early 2023, I built a couple of experimental dashboards to share some of my early work on the problem and began to develop a framework for helping engineers make trade-offs between quality, speed and price. Late last year, I began to collaborate with my friend George Cameron (who I met interning at Google together many years ago!) to build Artificial Analysis. I’ll be sharing more over the coming months - about what metrics matter most in AI, how developers should think about comparing AI models for scaling production applications, how we see our role in the industry evolving as the AI frontier moves forward. As we move into the next phase, our vision is to become the definitive source for data and insights in the AI industry. To achieve this, we're growing our team and hiring now for engineering and analyst roles. If you’re excited about what we’re building here, we’d love to hear from you. Finally - please follow Artificial Analysis on LinkedIn and Twitter to stay in the loop for our upcoming launches and analysis!
Very useful and well laid out site! Any plans to allow csv and/or pdf export for any of the data?
We're huge fans as you know Micah - Artificial Analysis has saved us tonnes of effort in LLM selection.
From Day 1, it's been my go-to website whenever a new AI model is launched. Nice work Micah Hill-Smith George Cameron!
Micah Hill-Smith Love this! So excited for what’s to come for Artificial Analysis, and a pleasure to be part of the journey!
Micah Hill-Smith it’s incredible to see how far Artificial Analysis has come in a short 6 months! It’s been such an amazing time working with you and the team - I’m excited to see what comes next.
It has been a great journey co-founding Artificial Analysis with you Micah! Still a long way to go! 🚵♂️
Impressive 6 months! Keep killing it Micah Hill-Smith
Micah Hill-Smith crushing it, can't wait for our podcast chat next week on your journey!!!
Love this, Micah - it's great to see you doing so well!
Co-Founder & CEO at Artificial Analysis | Ex-McKinsey
1moAll-In Podcast #1 - E165 (52:00 to 1:12:00): https://youtu.be/FHO4hoXc75k?si=R1HElXslVmcjiwpF&t=3162 Job Description (Senior Software Engineer) - please get in touch! https://docs.google.com/document/d/12FK2IiKXz9LOPUdrhPiPiIciKWM_PsFlSMw1UbWFbcc/edit?usp=sharing All-In Podcast #2 - E167 (28:20 to 31:30): https://youtu.be/z6vrKA_L5pk?si=iQiKoKZwYTxrPLda&t=1700 Andrew Ng: https://www.linkedin.com/posts/andrewyng_model-api-providers-analysis-artificial-activity-7215423883938516992-8I9Y?utm_source=share&utm_medium=member_desktop Latent Space Podcast: https://youtu.be/9xUTOwYcavo?si=kjwSYCB7TeLHoCvK&t=1559 SemiAnalysis: https://www.semianalysis.com/p/openai-is-doomed-et-tu-microsoft, https://www.semianalysis.com/p/groq-inference-tokenomics-speed-but VentureBeat: https://venturebeat.com/ai/sambanova-unveils-new-ai-chip-to-power-full-stack-ai-platform/ Groq: https://wow.groq.com/artificialanalysis-ai-llm-benchmark-doubles-axis-to-fit-new-groq-lpu-inference-engine-performance-results/ Artificial Analysis Website: https://artificialanalysis.ai/ Artificial Analysis Twitter: https://x.com/ArtificialAnlys