Fluree’s Post

A year ago, Denny Vrandečić of the Wikimedia Foundation, one of the key people behind Wikidata, made some telling remarks about when not to use LLMs during a keynote at the The Knowledge Graph Conference: “Why would you ever use a 96 layer LLM with 175 billion parameters to generate a multiplication, which is a single operation with a CPU? Just because an LLM can do these kinds of things doesn’t mean they should be. Why should you be generating knowledge again and again when you can just look it up in a confident way?…. It’s just not very efficient.” What's happened in the past year since Vrandečić made this observation? Platform providers such as Fluree are making it possible to harness the power of LLMs and knowledge graphs together in ways that allow more accuracy, efficiency and security. https://t.ly/MyPQ8

Super excited to share some of the techniques we are using for efficient LLM inference! Many thanks to Daya Khudia and the team for this hard work. Some quick things to know: 1) Use a good server like vllm or TRT-LLM 2) Good servers have continuous batching which will group incoming requests together on a per-token level to balance latency and throughput 3) At lower batch sizes, inference latency is determined primarily by the amount of memory bandwidth. At high batch sizes it is mainly a factor of FLOPS. 4) Mixture of Expert models are awesome because fewer parameters are in use at a given time, reducing demands on memory bandwidth and FLOPs. 5) Quantization helps with little accuracy loss since it can cut memory bandwidth in proportion to the quantization size. Also FP8 on newer cards can double the effective FLOPs! https://lnkd.in/gszuGAjz

Looking for a vector database that stands out from the rest? Look no further! Our #vectordatabase offers unparalleled speed and efficiency. Here's how we're different: 1. Search text, relational, and #unstructured data 7-20x faster than OpenSearch 2. Compress time data to 1% the size of other vector databases 3. Unique time search uses fractional memory (19 MB vs. 2.4 GB), no embedding needed 4. Unlimited time windows for retrieval vs. minutes for competitors 5. All benefits obtained using CPUs (no need for specialized – and expensive – GPUs) Get in touch with us if you want to learn more! #genai #largelanguagemodels #vectorsearch #vectordb https://lnkd.in/ge3qYbUc

If you are building #RAG applications, the first step is to create embeddings from your data. Checkout how you can do this with #Ray and Anyscale at scale and with unprecedented efficiency (90% cost reduction 🤯 vs. #OpenAI embedding API). If you are interested in trying this, please fill out https://lnkd.in/gDnvqZvR This is achieved through Ray's *unified* support for all steps of the pipeline with full *flexibility* on hardware / ML model for each operation (e.g. a mix of CPU, A10, and A100). 🚀 https://lnkd.in/gFeZ5akX (bit.ly/rag-embedding) Great collaboration with Pinecone team (check out their major launch https://lnkd.in/guYY7KPF!). Nathan Cordeiro Ram Sriharsha Roy Miara ❤️

The Power of Indexing in Context Searching

Over the last few months, I've been speaking to many customers around LLM Governance. The reality is long term, organizations will need implement custom guardrails specific to their use cases and industries in addition to standard off the shelf guardrails(e.g. hate speech detection, violence, etc). NVIDIA's NeMo Guardrails is an excellent library to achieve this. And you can pretty easily deploy this as middleware in Snowflake Snowpark Container Services wrapping our Cortex LLM functions which provide easy, instant access to a large library of both closed source and open source LLMs. Checkout how to build and deploy such an architecture here 👇🏻: https://lnkd.in/eXWt6MXe

A very short summary of what is RISC-V. ⌨ 👁️🗨️ https://lnkd.in/eXCVgEKm