Robin Linacre’s Post

Indexing large datasets (100M+ embeddings) requires a lot of memory in most vector databases: For 100M documents/embeddings, most vector databases require about 500GB of memory, driving the cost for your servers accordingly high. This repository offers methods to be able to search on very large datasets (100M+) with just 300MB of memory, making semantic search on such large datasets suitable for the Memory-Poor developers.

📐 Vector Search Optimization via KMeans, Voronoi Cells and Inverted File Index (aka “Cell-Probing”). I had quite a lot of fun 🤩 understanding how to improve vector search without impacting OLTP performances. 😯 As usual, a divide-and-conquer plus low-friction provisioning offered by containers to spin up a compute node was the solution. 🚀 Here's what I did, so that you can do it too, no matter which version of #SQLServer or #AzureSQL you are using. It is fun, you learn a lot and it is practical. The article also shows quite clearly all the reasons why *I love* 😍 being a developer: lateral thinking, creativity, and engineering all together. Fantastic! 💪

You can now easily run graph queries on #apacheiceberg tables! PuppyGraph, a Tabular partner, just came out of stealth mode 🙌 🙌 🙌 , and their unique graph query engine is already supported in Tabular. Read the blog here, and try it out using our free tier! https://lnkd.in/dEAW5cxJ

The Best Vector Databases for Storing Embeddings

Optimizing Performance with FastAPI

Why are our new MF4 decoders awesome? 🔥 🔍 We recently added a new free software tool for the CANedge, which we call 'MF4 decoders'. These are simple executables that let you DBC decode your raw CAN/LIN data into CSV or Parquet files via drag & drop (or CLI). Pretty basic, right? 🌫 However, what this does is to provide a simple, consistent and efficient method for creating Parquet data lakes - with zero coding required. This includes optimized folder structuring, partitioning, file concatenation and more - non-trivial stuff. "OK, but what's so great about Parquet data lakes?" Well, quite a lot - and more than I can explain in this post. However, I did create a visual to illustrate one core feature: You can store your data lake anywhere - and integrate with everything. ☁️ Parquet data lakes are simply a bunch of folders containing binary files - exactly like our MF4 log files from the CANedge. This means you can store them locally, on S3, Azure blob storage, Google Drive etc. - and it makes storing that data up to 95% lower cost vs. databases. ⚙️ Further, Parquet files are directly supported in a wide array of tools such as #PowerBI, #Python, #MATLAB and more. Compared to MF4 files, Parquet files are probably >100x more widely supported. 🔗 But the true magic happens when you combine Parquet data lakes with various 'interfaces' such as Amazon #Athena, Google #BigQuery, Azure #Synapse, #DuckDB, #ClickHouse and more. These tools essentially super charge your data lake queries, boosting performance and speed to extreme levels. Further, they expose your data lake via standard SQL, literally exploding the number of possible integrations. We will deep dive on many of these integrations over the coming months - starting with #Grafana and #Excel. But I recommend to already now check out our article on the MF4 decoders below and to try them out on your own data - any feedback is encouraged! https://lnkd.in/dj7xJ8us #canbus #parquet #datalake #j1939 #telematics Amazon Web Services (AWS) CAN in Automation (CiA) Grafana Labs DuckDB ClickHouse MathWorks

Week #6 of building a #storage service Past few weeks i discussed about object persistence, replication and fault tolerance. So, before I investigate the nuances and technical difficulties that each tier in the storage service presents, the plan is to roll out a completely working MVP. This week's focus is search. I was looking out for products that provide search as a feature on the stored objects. There were a bunch of products which had family integrations like s3 with elastic search. Similarly with other service providers like #gcp and #azure. The goal here shifts slightly, our object storage needs to support integration with these external services now. Putting aside my slight bias for a unified theory where everything is built into the product, I'm taking some time to explore a couple of services and how they can be integrated with our Object store. #Meilisearch - Meili search is a rust based search offering. Like always here is my #kta aka key take aways from this exploration. 1. You can easily load your JSON documents onto meili search and it gives you out of the box semantic search with a nice UI. 2. When you load your documents, Meili Search refers to this process as creating an index. For instance, if you have a million JSON docs related to movies, you can create an index for them in a matter of seconds. 3. Meili creates multiple in memory/persistent data structures: i. Inverted Indexes: These maps each word in a document to a set of documents. ii. Finite State Transducers: Utilized for string mapping operations. iii. Rtrees: These are used for effective spatial and multi-dimensional search capabilities, enhancing search efficiency for location based data. iv. Query Graphs: Meili Search generates all possible variations of user queries along with possible typo's. 4. Few Notable Algorithms: i. Levenshtein Algorithm - For edit distance calculations. ii. Bucket sort - Sorts documents based on some ranking rules Here's the link to the deep dive of their engine. https://lnkd.in/gU9DgesC Discoveries for future learning http://www.lmdb.tech/doc/ #computerscience #storage #database #algorithms #datastructures #product #software #softwaredevelopment

Big day for single-node analytical engines! Polars 1.0.0 alpha release and MotherDuck General Availability were announced today. There is a big reason why this is important - more on that in a bit. First, a quick recap on both; 𝐏𝐨𝐥𝐚𝐫𝐬 Is a DataFrame library implemented in Rust for high-performance local OLAP workloads. It was created from the need to replace Pandas as a performant DataFrame API for Python and has evolved to operate on top of the data lakehouse, with support for Delta and Iceberg. 𝐌𝐨𝐭𝐡𝐞𝐫𝐃𝐮𝐜𝐤 Is a serverless platform to run 𝐃𝐮𝐜𝐤𝐃𝐁 databases - the “SQLite for Analytics” written in C++ - on the cloud. MotherDuck’s General Availability follows DuckDB’s 1.0.0 release last week, making them both ready for production use cases. Why is this big? Single-node analytical engines such as #Polars and #DuckDB have been proving that we can run highly efficient analytics without a distributed framework. And when you place that on top of modern hardware, it’s getting harder to find real-world use cases that can justify spinning up a Spark cluster. The issue until now, however, is that Spark is widely adopted and battle-tested. It’s been the dominant, trusted choice for use in production. With Polars and DuckDB + MotherDuck going production-ready, their use cases and adoption can vastly expand. That’s especially great, being both open-source. I wouldn’t be surprised to see more and more successful stories of distributed to single-node analytics migrations showing up! It can be a true paradigm shift long-term. What do you think? #dataengineering

🚀 Spice OSS v0.14.1-alpha (Jun 24, 2024) is now available! Spice OSS focuses on three problems when building data and AI-driven apps. 1. Access and query data wherever it lives. 2. Making that data fast and cost-effective to be useful. 3. Making it easy to use that data with machine learning models and AI. The v0.14.1 release includes: 🌟 Enhanced Performance & Data Integrity: Primary key and unique index constraints, plus faster queries using indexes on frequently filtered columns. 🔗 New Data Connectors: GraphQL data connector and improved dataset registration. 🛡️ Improved Stability & Quality: Query result caching to handle bursts, query history logging, and better federated query push-down. 📈 Telemetry & Metrics: Detailed runtime metrics, query duration, failure tracking, and a comprehensive Grafana dashboard for monitoring. 🗂️ Filesystem & Database Support: Directly use files as data sources and leverage better support for PostgreSQL, DuckDB, and GraphQL connectors. Check out the release on GitHub at https://lnkd.in/gfw8zSdn. #spiceai #data #developers #performance #postgres #duckdb #graphql 🔥

Setting Up Your Qdrant Vector Database via #TowardsAI → https://bit.ly/3JNpCMN