Meilin Xu’s Post

Anyone in for an early Christmas present? Snowpark Container Services is now in public preview! It allows you to bring any workload in any language any runtime any code to run right next to your data - and even power the compute via CPUs or GPUs. This makes Snowflake the ideal home for a whole host of new workloads: LLMs, Model Training in Data science, Data Applications and even more advanced Data Processing and Data Engineering. https://lnkd.in/eXSwK__S

If you are building #RAG applications, the first step is to create embeddings from your data. Checkout how you can do this with #Ray and Anyscale at scale and with unprecedented efficiency (90% cost reduction 🤯 vs. #OpenAI embedding API). If you are interested in trying this, please fill out https://lnkd.in/gDnvqZvR This is achieved through Ray's *unified* support for all steps of the pipeline with full *flexibility* on hardware / ML model for each operation (e.g. a mix of CPU, A10, and A100). 🚀 https://lnkd.in/gFeZ5akX (bit.ly/rag-embedding) Great collaboration with Pinecone team (check out their major launch https://lnkd.in/guYY7KPF!). Nathan Cordeiro Ram Sriharsha Roy Miara ❤️

OCI GPU Clusters backed by high performance file solutions to feed them are a potent combo!

You often hear me speak about domain-specific computing and why it is so critical in reconstructing search to support modern real-time data retrieval. I’ll start by emphasizing again that legacy software-based search solutions have long reached their glass ceiling and are not able to support real time search at a billion-scale without compromise of price, speed, or relevancy. The reason domain-specific computing is so powerful is that it skips the standard software semantics, cache hierarchy, and other CPU abstractions. It then implements the core parts as a custom datapath processor. Together with a new software stack, this runs search and information retrieval workloads hundreds of times faster. Furthermore, Hyperspace Cloud processing unit includes tens of dedicated search cores running proprietary instruction sets to filter, rank, and aggregate search results in a super-efficient way. These custom search instructions, along with advanced data prefetching, enable speeds that can’t be matched by general-purpose CPUs. Interested in learning more about how you can shatter the limits of your search? https://lnkd.in/dGVtdfC2 #elasticsearch #dataretrieval #vectorsearch #genai #llm #keywordsearch #lexicalsearch #database.

Read this detailed blog on how GPU optimization works for training, fine-tuning, or inference for LLMs, or LMMs.

Investing $10M & 2months, Databricks releases an OSS MoE (16 total) GenAI model in DBRX (Base / Instruct) that outperforms current OSS alternatives that can be privately deployed using MosaicML (and 4x H100 GPUs w/320GB of RAM) - no image as of yet & same restrictions on use as Llama (700M users).

The engineer in me couldn't resist running the numbers since I have always been curious about running LLMs on very low latency, high throughput, and minimal GPUs. As the paper mentions, BitNet b1.58 matches 16-bit LLM baselines. The existing alternative to this, 4-bit quantization, still looses on precision and performance of the LLM. If this becomes a reality while maintaining precision, it could truly revolutionize the game. For a 7-billion parameter language model (LLM) like Mistral or Llama: Total Memory = Number of weights × Precision size Total Memory Requirement: For 32-bit precision = 7 billion × 4 bytes = 26+ GB For 1-bit precision = 7 billion × 1/8 bytes = 0.8+ GB

Super excited to share some of the techniques we are using for efficient LLM inference! Many thanks to Daya Khudia and the team for this hard work. Some quick things to know: 1) Use a good server like vllm or TRT-LLM 2) Good servers have continuous batching which will group incoming requests together on a per-token level to balance latency and throughput 3) At lower batch sizes, inference latency is determined primarily by the amount of memory bandwidth. At high batch sizes it is mainly a factor of FLOPS. 4) Mixture of Expert models are awesome because fewer parameters are in use at a given time, reducing demands on memory bandwidth and FLOPs. 5) Quantization helps with little accuracy loss since it can cut memory bandwidth in proportion to the quantization size. Also FP8 on newer cards can double the effective FLOPs! https://lnkd.in/gszuGAjz

A year ago, Denny Vrandečić of the Wikimedia Foundation, one of the key people behind Wikidata, made some telling remarks about when not to use LLMs during a keynote at the The Knowledge Graph Conference: “Why would you ever use a 96 layer LLM with 175 billion parameters to generate a multiplication, which is a single operation with a CPU? Just because an LLM can do these kinds of things doesn’t mean they should be. Why should you be generating knowledge again and again when you can just look it up in a confident way?…. It’s just not very efficient.” What's happened in the past year since Vrandečić made this observation? Platform providers such as Fluree are making it possible to harness the power of LLMs and knowledge graphs together in ways that allow more accuracy, efficiency and security. https://t.ly/MyPQ8

Really interesting work by the folks at Voltron Data on Theseus, their new OLAP GPU query engine for running on distributed GPU clusters. Turns out that for truly insanely large workloads, like somewhere between 30TBs to 100TB per query, making optimal use of GPUs is more cost-effective than Spark - plus, the overhead of dealing with 100+ nodes clusters. What we call Big Data today is very different than what we used to 10 years ago. 95% of enterprises never see this kind of loads, but it's still...really cool. I really enjoyed the write up with all the benchmarks (and the conversion from Spark costs to Big Mac 😄 ). https://lnkd.in/dpt9EmjV