Zhe Zhang’s Post

Investing $10M & 2months, Databricks releases an OSS MoE (16 total) GenAI model in DBRX (Base / Instruct) that outperforms current OSS alternatives that can be privately deployed using MosaicML (and 4x H100 GPUs w/320GB of RAM) - no image as of yet & same restrictions on use as Llama (700M users).

Datadog’s Container Overview dashboard now gives you out-of-the-box visibility into CPU, memory, and network metrics from containers running OTel-instrumented applications. Learn more about this and other features announced at DASH 2023: https://lnkd.in/ehRu5SKg Datadog #monitoring #observability #innovation

Looking for a vector database that stands out from the rest? Look no further! Our #vectordatabase offers unparalleled speed and efficiency. Here's how we're different: 1. Search text, relational, and #unstructured data 7-20x faster than OpenSearch 2. Compress time data to 1% the size of other vector databases 3. Unique time search uses fractional memory (19 MB vs. 2.4 GB), no embedding needed 4. Unlimited time windows for retrieval vs. minutes for competitors 5. All benefits obtained using CPUs (no need for specialized – and expensive – GPUs) Get in touch with us if you want to learn more! #genai #largelanguagemodels #vectorsearch #vectordb https://lnkd.in/ge3qYbUc

Microsoft just released a paper on splitwise. An optimization tooling for LLM inferencing and therefore reducing the amount of GPU's required. In short: machines are maintained in different pools and dedicated to the two distinct LLM inference phases. The mixed pool grows and reduces according to runtime demand. KV-cache encompassing the state of the query after the prompt phase is transferred from the prompt machines to the token machines over InfiniBand with very low latency. Check out the blogpost to learn more: https://lnkd.in/e7QX4BVx Alan Weaver Kathryn Jesaitis Papandrew Alex Zeltov Eyas Taifour Răzvan Tănase

OCI GPU Clusters backed by high performance file solutions to feed them are a potent combo!

In the latest Engineering Blog post by Prodigy Education, Staff Site Reliability Engineer Erik Krieg takes a look at how to run a GPU-accelerated open-source Large Language Model (LLM) inference workload using Elastic Kubernetes Service (EKS). https://lnkd.in/geSPErxG #insideprodigy #largelanguagemodels #engineeringblog

#Topics PyTorch/XLA 2.4 improves Pallas and adds “eager mode” [ad_1] And now, instead of having to call xm.mark_step() you can call torch_xla.sync()instead. These improvements make it easier to convert your code over to PyTorch/XLA and improve the developer workflow. For more changes to API calls, check out the release notes. Experimental eager mode If you’ve been working with PyTorch/XLA for a while, you know that we refer to models being “lazily executed.” That means that PyTorch/XLA creates the compute graph of operation before sending models over to be executed on the XLA device target hardware. With new eager mode, operations are compiled and then immediately executed on the target hardware. The catch to this feature though is that TPUs themselves do not have a true eager mode, since each instruction is not sent to the TPU by default right away. On TPUs, we achieve this by adding a “mark_step” call after each PyTorch operation to force the compilation and execution. This results in the functionality of eager mode but as an emulation rather than as a native feature.  Our intent with eager mode in this release is not to run it in your production environment, but rather in your own local environments. We hope that eager mode makes it easier to debug your models ...

Eduardo Alvarez does well in this article. I especially like his attention to scalability, fidelity and latency. Using RAG we can keep a focus on the data pipeline. Additionally the pre and post processing retrieval offloads the work to CPU for more responsive actions (serving a narrow audience over the billions served). Utility via CPU infrastructure is a unique take. The examples are good and the clarity includes the vector embed in an easy to understand walk through.

Lilypad is currently free to use and lets you run your own containerised workloads using connected GPUs for heavier compute processing. Here's a blog post I wrote on how to get started with your own job modules optimised to run on the Lilypad Network. https://lnkd.in/gg4TdqsK

Have you wondered how can you optimize your compute for building LLMs? Apart from the model architecture and dataset, the other most important component of building LLMs is - infrastructure (GPUs, TPUs, or LPUs)! To train, fine-tune, or run inferencing on your models there are extensive compute requirements along with costs. Hence it is critical to optimize the infrastructure to support your workloads: https://lnkd.in/dJFPwwJz I have authored a detailed blog on how GPU optimization works for training, fine-tuning, or inference for LLMs, or LMMs. PS: At this point, most models being built are multimodal, hence I like using the term LMMs instead of LLMs. If you found the article insightful, share this with your network ♻️ ------------------------------------------------- If you don't want to miss any of my posts, go to my profile (Aishwarya Srinivasan) and click on the 🔔 to get notified about all my posts ❤️