At #ISCA2024 we will explain how we co-designed a compilation-based software stack and a class of accelerators called LPUs, which result in high utilization and low end-to-end system latency. We'll talk about the challenges of breaking models apart over networks of LPUs, and outline how this HW/SW system architecture keeps enabling breakthrough LLM inference latency at all model sizes. https://hubs.la/Q02BrFnL0
Groq’s Post
More Relevant Posts
-
Brute force AI accelarator scaling is not sustainable. Bigger dies, more transistors, more HBMs are not going to get us the 10x perf/year genAI demands. By 2027 AI is projected to consume more power than all of the country of Argentina (hosting ISCA). I am excited to present Groq’s LPU Inference Engine - a vertically optimized genAI inference system that uses 30x less devices to achieve 6x better perf, 4 lower cost, and 3x lower energy consumption. Let’s make every transistor & electron count.
At #ISCA2024 we will explain how we co-designed a compilation-based software stack and a class of accelerators called LPUs, which result in high utilization and low end-to-end system latency. We'll talk about the challenges of breaking models apart over networks of LPUs, and outline how this HW/SW system architecture keeps enabling breakthrough LLM inference latency at all model sizes. https://hubs.la/Q02BrFnL0
-
-
Did you miss last week’s virtual SNIA Compute, Memory, and Storage Summit? The UCIe Consortium session “Enabling an Open Chiplet Ecosystem with UCIe” is now available on demand with FREE event registration. During the session, Brian Rea and Richelle Ahlvers (Intel Corporation) introduce new features in the UCIe 1.1 specification and explore how UCIe benefits the storage industry. Watch the full presentation: https://bit.ly/4ecdTpl #UCIeConsortium #UCIe #chiplets #SNIA #SNIACMS #storage
-
At #GTC24, learn about how the #GraceCPU and #GraceHopper Superchips are built to solve the largest computing problems. Register for these GTC sessions today: https://nvda.ws/4aRWsZo
-
-
Assistant Professor at NTU Singapore. Cloud and serverless computing, Systems for ML
At NTU and UoEdi, we believe the future of #LLM systems is #serverless. Large-scale LLM inference serving systems, bound by power and chip shortage, require new highly efficient designs. To achieve this, our OSDI'24 paper brings serverless autoscaling to LLM workloads. #vHive
Our work on enabling cost-effective serverless inference for LLMs got accepted into #OSDI24! This work features a new LLM checkpoint format boosted by a fast loader over multi-tier storage, LLM inference live migration, and novel GPU serverless architectures! This work is led by Yao Fu (his 2nd OSDI paper since started as a Ph.D. in 2021). Joint work with Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, and Yuvraj Patel. Congratulations on the team! #LLMs #Serverless #Research
-
-
In order to optimize the operational efficiency of LAMs, this week we are exploring the use of LoRAX (LoRA Exchange) - we're particularly drawn to its ability to optimize GPU utilization and provide scalability for fine-tuned model inferencing. LoRAX allows users to serve 1000s of task-specific models into a single GPU, significantly reducing expenses associated with serving multiple models. This is achieved through a combination of dynamic adapter loading, tiered weight caching, and continuous multi-adapter batching. This allows seamless management and operation of multiple fine-tuned models, minimizing technical overhead and maximizing efficiency. We are planning to use LoRAX to serve nearly 10+ adapter weights using just a single server which is horizontally scalable based on increased load dynamically on the go. This will help software applications to have different multi-tenant models for each user efficiently.
-
-
As SC23 concludes, our CEO Bjorn Kolbeck shares his thoughts about our latest enhancement: Quobyte's RDMA feature. This addition is pivotal in making HPC storage more accessible and performance-driven. In the video, he talks about how integrating RDMA into our storage platform will improve user experience, offering lower latency in diverse infrastructures. #Quobyte #RDMA #HPC #SC23 #Innovation #AccessibleHPC #HPC
-
The training runtime for a LLaMA2 LLM model with 7 billion parameters across different GPU configurations on a single 32 GPU SuperNODE. SuperNODE offers a cost-effective, scalable solution for demanding computing environments. https://bit.ly/3xKBnAV #GenAI #GigaIO #LLM
-
-
Efficient data movement, real-time responsiveness and low latency are the key fundamentals of MIPS differentiation and giving customers the Freedom to Innovate Compute. Hear Sameer Wasson's vision for the future of MIPS. #FreedomtoDesign #FreedomtoInnovate https://lnkd.in/dDH5qP-i
MIPS: Freedom to Innovate Compute
https://www.youtube.com/
-
Learn about the outstanding deficiency in current RTOS architectures: The Diminishing Bandwidth Problem I have 2 patents that solve this problem, thereby immediately rendering all current RTOSes obsolete. See this easy-to-understand presentation: https://lnkd.in/gG-6nxSQ
