Jon S.’s Post

While Nvidia's earnings are making headlines, a new type of processor called an "LPU," which stands for Language Processing Unit (as opposed to a GPU), is being offered by a company named Groq, founded by a person who co-founded and worked at Google's TPU hardware lab from 2013-2015. Groq is offering API access at half the cost and 4 to 5 times the speed of other API providers. Their LPU accelerator cards are around $20K (https://lnkd.in/geaTtNVB). I'm unsure how long the cheap API access will last. With the right setup, inference can be up to 18x that of GPUs. While not ideal for training new models or updating existing models due to limited memory, Groq stands out for its ability to run pre-trained models far more efficiently than with standard hardware. Groq is an ASIC chip with a matrix engine similar to a systolic array. It uses FP16 for calculating all activations, while some data is stored as FP8 at rest. It is a deterministic system that does not require data parallelism to maximize throughput and inference batching can be very minimal or even avoided. Service providers like Together.ai have to manage batching to achieve optimal throughput. For example, they may batch 100 requests in one operation to fully leverage their hardware. This increases throughput at the cost of potentially higher latency for individual requests. This is not a problem for Groq, not even with their current 14nm chips (they are working on 4nm now). We're currently experimenting with Groq using Mixtral 8x7b and the performance is amazing. Interesting tidbit: I believe X.ai (Elon Musk) could be sued by Groq for trademark infringement, as Groq and Grok are confusingly similar. The Groq trademark was filed in 2017 whereas X filed for Grok in 2023. I've never seen a better cease & desist letter: https://lnkd.in/grk6jjGe Groq demos: https://lnkd.in/gnpGzD_r https://lnkd.in/ghsMg654 Demo: https://groq.com Another very interesting demo: https://vapi.ai #llm #ai #groq #mistral #googlegemini #openai #chatgpt #anthropic

IBM Storage Fusion HCI System has released a new GPU server today that takes a huge step forward in the support of AI applications. The new server, called the G03, can be configured to have as many as 8 double-wide GPU adapter cards.  Fine-tuning and inferencing with foundation models used in AI applications increasingly requires faster and more powerful hardware, and the Fusion HCI G03 meets that need head on! #ibmstorage #ibmstoragefusion #storagefusionhci #gpu #openshift https://lnkd.in/gs5pdza7

Selecting the best hardware for performance and integration is key for data processing. Enter the #GPU - SQream tested and certified the L40S and H100 GPUs to understand how they accelerate and scale with #data operations. Interested in the insights we found? Read our latest blog --> https://bit.ly/47jzjNa

Maximize efficiency for inferencing with AI on #HPE ProLiant servers with NVIDIA GPUs! The HPE ProLiant DL380 bundle is designed to offer a complete AI solution that delivers higher performance per dollar. 🤖🙌 Check out this solution brief to learn more—and reach out to CompuTech if you have any questions.

Maximize efficiency for inferencing with AI on #HPE ProLiant servers with NVIDIA GPUs! The HPE ProLiant DL380 bundle is designed to offer a complete AI solution that delivers higher performance per dollar. 🤖🙌 Check out this solution brief to learn more—and reach out to CompuTech if you have any questions.

Maximize efficiency for inferencing with AI on #HPE ProLiant servers with NVIDIA GPUs! The HPE ProLiant DL380 bundle is designed to offer a complete AI solution that delivers higher performance per dollar. 🤖🙌 Check out this solution brief to learn more—and reach out to Consiliant Technologies LLC if you have any questions.

The (right) Software makes Hardware better. Many years ago I worked at Sun Microsystems which at the time had the most performant technical desktop and servers in the world, powering all the large banks and big technology platforms like Ebay. Sun not only created their own RICS-based CPUS (UltraSPARC), but also their own servers and software. They did this because the marriage between hardware and software had the most impact on customer success - that is software tuned to run optimally on specific hardware performed much better than generic off-the-shelf or open source bits. I thought about this when an article by Nvidia (https://lnkd.in/dqx8hP5y) caught my eye because it reminded me of the importance of using software which helps you get the most out of hardware. Whether benchmarking performance, maximizing utility, speed, TCO, or customer experience, the right software makes all the difference. In the age of rapidly advancing technologies such as the field of Generative-AI where GPUs, Frameworks, LLM, RAG databases and many more components are being created at breakneck speeds, ultimate performance and ultimate value will inevitably depend on using the right libraries, tools, SDKs and models that are tuned to maximize the performance of the underlying hardware.

Faster, Longer Lasting SSD Performance: How Graid Technology is Changing the Game with GPU-Accelerated RAID 🌐 "Since launching in 2021, Graid Technology has experienced explosive expansion, projecting over 475% year-over-year revenue growth from 2022 to 2023. Strong customer interest across verticals like media and entertainment, telecom, oil and gas, finance, healthcare, and more validate SupremeRAID™’s game-changing value proposition." Curious how we're transforming the industry?💡💎 SupremeRAID™, designed to eliminate storage bottlenecks and unleash the full potential of high-speed NVMe SSDs, achieves unprecedented performance by offloading intensive parity calculations from the CPU to the GPU. Learn how in the full article from VMblog: https://zurl.co/eNH6 #VMBlog #media #RAID #GPU #storage #SMCI #NVMe #NVMEeoF #SSD

Virtual Machine SKU’s and how to read them. Yesterday I shared a post explaining the different types of (most populair / used) VM families available and how they are categorized (see: https://lnkd.in/eKEucgQv). Today we’ll continue along those lines, but we’ll take it a step further and dissect the “code” by which Virtual Machines on Azure are listed. Even if you have been working with Azure for a while it can still be confusing. Let’s have a look…   If we take a D16ds_V5 machine, what does that all mean?   D = Part of the Standard VM family. 16 = 16 CPU cores d = Has a Local temp desk s = Compatible with Premium SSDs V# = Gen Version of the machine   And that’s just one example. There are other families (as mentioned in yesterday’s post), AMD powered machines, CPU constraint machines, and other (older) gen versions as well.   Here at Nerdio we try to hide complexity, in general as much as we possibly can. We are all about making things simple, though sometimes the only thing you can do is try to explain it as straightforward as possible and help people on their way, by providing a simple cheat sheet, for example like the one below.   Hope this helps. 

Benchmark tests completed. The #NVIDIA L40S & H100 GPUs are certified on SQream for supercharged Data & Analytics! The SQream platform harnesses the speed and power of NVIDIA's accelerated compute for big data, enabling... - faster analytics / reports / BI - accelerated data ingest/prep/ETL - optimized data pipelines for AI/ML Game changing technology!