Davide Mauri’s Post

Caching++🗃️ I'm excited to share a new caching feature that we've been working on in letsql. This feature allows you to cache the results of compute expressions (`ibis.expr`), whether in the upstream engine or on local disk, making your iterations faster and more efficient. ⚡️ Why does this matter? 🤔 As data scientists, we often face the challenge of having to repeatedly pull data from its source or re-compute results during the iterative process of building machine learning models. This not only slows down our workflow but also puts unnecessary strain on source systems and network bandwidth. Our new caching feature addresses this by providing a seamless way to cache data, significantly speeding up your iteration cycles while reducing cognitive load. No more manual processes, no more disjointed workflows—just pure efficiency. How does it work? 🔍 With LETSQL, you can easily cache sub-expressions representing upstream queries i.e. `ibis.expr`. The data is then stored either in the upstream engine or locally, depending on your needs. The best part? This feature is designed to work with multiple engines, giving you the flexibility to choose the best storage option for your workflow. 👉 Ready to give it a try? Simply drop into an interactive iPython shell by running: 💻 nix run github:letsql/letsql We’re still in the beta phase, and your feedback is incredibly valuable as we refine this feature and work towards a stable API. Let us know what you think! 🔗 Learn more about the feature in our latest blog post: https://lnkd.in/eGpxwUaU #DataScience #MachineLearning #Caching #DataFrames

Indexing large datasets (100M+ embeddings) requires a lot of memory in most vector databases: For 100M documents/embeddings, most vector databases require about 500GB of memory, driving the cost for your servers accordingly high. This repository offers methods to be able to search on very large datasets (100M+) with just 300MB of memory, making semantic search on such large datasets suitable for the Memory-Poor developers.

Using Clojure with graph databases?

Learn how scalar quantization with half-precision vectors "halvec" reduces storage costs without compromising the performance and efficiency of pgvector indexes. All Neon Postgres databases come with pgvector https://lnkd.in/gcaD_mhD

Plan for today and tomorrow: scalar and binary quantization are coming soon to #pgvector and, when used appropriately, can provide both space/memory savings and performance boosts for managing vector data in #PostgreSQL! Learn more: https://lnkd.in/ebfuPHT3

𝗔𝗩𝗟 𝗧𝗿𝗲𝗲𝘀 𝗗𝗮𝘁𝗮 𝗦𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 AVL Tree is a self-balancing binary search tree. The key feature of an AVL tree is that the heights of the two child subtrees of any node differ by at most one. This feature of AVL Tree is called the balance factor and it solved the problems of Binary Search Trees. 𝐖𝐡𝐚𝐭 𝐀𝐕𝐋 𝐓𝐫𝐞𝐞𝐬 𝐡𝐚𝐯𝐞: • Binary Search Tree Property: Every node in the tree follows the binary search tree property, where the left subtree of a node contains nodes with keys less than the node's key, and the right subtree contains nodes with keys greater than the node's key. • Balance Factor: The balance factor of a node is the difference between the heights of the left and right subtrees of that node. For an AVL tree, the balance factor of any node must be 0 or 1. This property ensures that the tree remains approximately balanced, keeping the operations efficient. • Insert operation on AVL Trees: If an insertion or deletion operation causes the balance factor of any node to become greater than 1, the tree performs rotations to restore balance. These rotations are: - Single Right Rotation - Single Left Rotation - Left-Right Rotation - Right-Left Rotation In the gif image below, we compare AVL Trees with unbalanced binary search trees. In an unbalanced Binary Search Tree, the worst-case time complexity for operations like insertion, deletion, and search can be O(n), where n is the number of nodes in the tree. This is because the tree can degenerate into a linked list in the worst case, leading to performance issues. 𝐀𝐝𝐯𝐚𝐧𝐭𝐚𝐠𝐞𝐬 𝐨𝐟 𝐀𝐕𝐋 𝐓𝐫𝐞𝐞𝐬: Because the tree is balanced, the height is kept to O(logn), ensuring efficient search operations and suitable for applications where the data is frequently updated. 𝐖𝐡𝐞𝐫𝐞 𝐀𝐕𝐋 𝐓𝐫𝐞𝐞𝐬 𝐢𝐬 𝐮𝐬𝐞𝐝: • AVL Trees are often used in Databases and File systems for indexing and efficient data retrieval in databases, such as MySQL and PostgreSQL. • AVL trees can be used in memory management systems for efficient allocation and deallocation of memory blocks, ensuring fast lookup and retrieval of free memory regions, and many more usage... #datastructure #binarysearchtree #AVLtree #algorithm #softwaredevelopment #wecommit100xshare

💡 Tuesday tech tip: Materialized Views are a global index. Some common use cases for them are Indexing with denormalization, different sort orders, and filtering (pre-computed queries). This free #ScyllaDB lesson goes over examples, the #CQL syntax to use, and limitations: http://ow.ly/R77l50NaUg2 #TechTips #NoSQL

You can now easily run graph queries on #apacheiceberg tables! PuppyGraph, a Tabular partner, just came out of stealth mode 🙌 🙌 🙌 , and their unique graph query engine is already supported in Tabular. Read the blog here, and try it out using our free tier! https://lnkd.in/dEAW5cxJ

Big day for single-node analytical engines! Polars 1.0.0 alpha release and MotherDuck General Availability were announced today. There is a big reason why this is important - more on that in a bit. First, a quick recap on both; 𝐏𝐨𝐥𝐚𝐫𝐬 Is a DataFrame library implemented in Rust for high-performance local OLAP workloads. It was created from the need to replace Pandas as a performant DataFrame API for Python and has evolved to operate on top of the data lakehouse, with support for Delta and Iceberg. 𝐌𝐨𝐭𝐡𝐞𝐫𝐃𝐮𝐜𝐤 Is a serverless platform to run 𝐃𝐮𝐜𝐤𝐃𝐁 databases - the “SQLite for Analytics” written in C++ - on the cloud. MotherDuck’s General Availability follows DuckDB’s 1.0.0 release last week, making them both ready for production use cases. Why is this big? Single-node analytical engines such as #Polars and #DuckDB have been proving that we can run highly efficient analytics without a distributed framework. And when you place that on top of modern hardware, it’s getting harder to find real-world use cases that can justify spinning up a Spark cluster. The issue until now, however, is that Spark is widely adopted and battle-tested. It’s been the dominant, trusted choice for use in production. With Polars and DuckDB + MotherDuck going production-ready, their use cases and adoption can vastly expand. That’s especially great, being both open-source. I wouldn’t be surprised to see more and more successful stories of distributed to single-node analytics migrations showing up! It can be a true paradigm shift long-term. What do you think? #dataengineering

🔒 Want to work with vectors, LLMs and RAG but worried about security? Did you know that txtai has full Postgres integration (dense vectors, sparse vectors, content and graph)? This can be combined with standard row level security to limit what content a user can utilize for GenAI processes. Learn more at the links below. Article: https://lnkd.in/eFeFNgYK Postgres docs: https://lnkd.in/e9aFks5Y