Fluree’s Post

🚀 𝐃𝐚𝐭𝐚 𝐋𝐚𝐤𝐞𝐡𝐨𝐮𝐬𝐞: 𝐀𝐩𝐚𝐜𝐡𝐞 𝐈𝐜𝐞𝐛𝐞𝐫𝐠🚀 ✅ 𝐖𝐡𝐚𝐭 𝐞𝐱𝐚𝐜𝐭𝐥𝐲 𝐢𝐬 𝐀𝐩𝐚𝐜𝐡𝐞 𝐈𝐜𝐞𝐛𝐞𝐫𝐠? 🔺 It is a table format created in 2017 by Netflix’s Ryan Blue and Daniel Weeks, for massive analytic datasets. 🔺 It overcame challenges with performance, consistency, and many of the challenges previously stated with the Hive table format. 🔺 Became open source in 2018. ✅ 𝐖𝐡𝐚𝐭 𝐚𝐫𝐞 𝐢𝐭𝐬 𝐤𝐞𝐲 𝐟𝐞𝐚𝐮𝐫𝐞𝐬? 🔺 𝑺𝒄𝒉𝒆𝒎𝒂 𝑬𝒗𝒐𝒍𝒖𝒕𝒊𝒐𝒏 𝒂𝒏𝒅 𝑽𝒆𝒓𝒔𝒊𝒐𝒏𝒊𝒏𝒈 Iceberg supports schema evolution, allowing changes like adding, dropping, renaming columns, and updating column types without affecting query results or data consistency. It also provides versioning, enabling rollback to previous states. 🔺 𝑷𝒂𝒓𝒕𝒊𝒕𝒊𝒐𝒏𝒊𝒏𝒈 Iceberg offers hidden partitioning that abstracts the complexity, optimizing query performance by automatically selecting the most efficient partitioning strategy based on the query workload. 🔺 𝑨𝒕𝒐𝒎𝒊𝒄𝒊𝒕𝒚 𝒂𝒏𝒅 𝑪𝒐𝒏𝒔𝒊𝒔𝒕𝒆𝒏𝒄𝒚 Iceberg guarantees atomic operations and consistent reads through its design, which includes atomic commit protocols. This ensures that updates are all-or-nothing, preventing partial writes and maintaining data integrity. 🔺 𝑫𝒂𝒕𝒂 𝑳𝒂𝒚𝒐𝒖𝒕 𝒂𝒏𝒅 𝑰𝒏𝒅𝒆𝒙𝒊𝒏𝒈 Iceberg optimizes data layout and includes built-in indexing mechanisms, such as manifest files and metadata trees, which enhance query performance by pruning unnecessary data reads. 🔺 𝑻𝒊𝒎𝒆 𝑻𝒓𝒂𝒗𝒆𝒍 Iceberg supports time travel, allowing users to query data as it existed at any point in time. This facilitates easy analysis of historical data and recovery from accidental changes. 🔺 𝑰𝒏𝒕𝒆𝒓𝒐𝒑𝒆𝒓𝒂𝒃𝒊𝒍𝒊𝒕𝒚 Iceberg is designed to be compatible with multiple processing engines such as Apache Spark, Apache Flink, and Trino, making it versatile and easy to integrate into existing data infrastructure. ✅ 𝐑𝐞𝐥𝐢𝐚𝐛𝐢𝐥𝐢𝐭𝐲 & 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞 Iceberg is designed to handle massive tables, often containing tens of petabytes of data by: 🔺 𝑬𝒇𝒇𝒊𝒄𝒊𝒆𝒏𝒕 𝑺𝒄𝒂𝒏 𝑷𝒍𝒂𝒏𝒏𝒊𝒏𝒈: Iceberg enables rapid scan planning, eliminating the need for a distributed SQL engine to read tables or locate files. 🔺 𝑨𝒅𝒗𝒂𝒏𝒄𝒆𝒅 𝑭𝒊𝒍𝒕𝒆𝒓𝒊𝒏𝒈: It optimizes data reading by pruning data files using partition and column-level statistics, leveraging table metadata to filter out unnecessary data. #dataengineering #تونس_أفضل

Excited to share my latest Medium blog post on LSM Trees! In "LSM Trees -A deep dive into a fundamental data structure used in databases," I explore the inner workings of LSM Trees. From write amplification to compaction, we'll cover it all. Whether you're a developer, database enthusiast, or just curious about tech, head on and dive in! Any suggestions are appreciated. Happy coding! #LSMTrees #Databases #MediumPost

Multi-Modal Indexing System in Apache Hudi. One of the core technical design choices that separates Apache Hudi from other open table formats is Indexing. From its inception, Hudi has been heavily optimized to handle mutable change streams with varying write patterns. Indexing plays a critical role in dealing with these writes that deems faster updates. They help with faster upserts/deletes by quickly locating the records that needs to be updated. 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐇𝐮𝐝𝐢'𝐬 𝐌𝐮𝐥𝐭𝐢-𝐌𝐨𝐝𝐚𝐥 𝐢𝐧𝐝𝐞𝐱𝐢𝐧𝐠? Hudi's multi-modal indexing system redefines the design of an indexing subsystem for data lakes. 𝐇𝐨𝐰 𝐢𝐬 𝐢𝐭 𝐢𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐞𝐝? - By using Hudi's Metadata table - Hudi's metadata table is actually a single internal Hudi Merge-on-Read table - It consists of different types of indexes such as column stats, bloom filter, files, record-level, etc. as individual '𝘱𝘢𝘳𝘵𝘪𝘵𝘪𝘰𝘯𝘴' 𝐂𝐨𝐫𝐞 𝐃𝐞𝐬𝐢𝐠𝐧 𝐏𝐫𝐢𝐧𝐜𝐢𝐩𝐥𝐞𝐬 - Scalable metadata: Size of the metadata table can grow with larger updates. The table needs to scale metadata to TBs of size - ACID updates: index & table metadata must always be up-to-date & in sync with the data table. No partial writes - Fast Lookups: Indexes size can be huge. For certain queries to be efficient, scanning entire indexes should be avoided 𝐇𝐨𝐰 𝐝𝐨𝐞𝐬 𝐢𝐭 𝐢𝐦𝐩𝐫𝐨𝐯𝐞𝐬 𝐩𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞? ✅ File Listings: For large datasets, listing files is a huge bottleneck. With files index in a multi-modal index system, listing is drastically reduced Vs direct S3 listing ✅ Data Skipping: On the read-side, column_stats index (min, max val) can be used to skip irrelevant data files. This serves as a 2nd level pruning Vs reading each #Parquet footers directly ✅ Fast Upserts: Bloom_filter index stores the bloom filters of all data files, thereby avoiding the need to scan the footers of each data file. Record-level indexes helps locating records faster. Having worked on other open #lakehouse table formats, I understand specific pain points in terms of write latency and how a multi-modal indexing system can be extremely beneficial. Detailed read in comments. #dataengineering #softwareengineering

🚀 Unlocking Database Performance with Hash Indexing! 🚀 As data grows exponentially, optimizing database performance becomes crucial for efficient data retrieval and management. I'm excited to share my latest article on Medium where I dive into the world of data structures, focusing on the powerful concept of Hash Indexing. 🔍In this article, you'll discover: 1. The basics and importance of hash indexing 2. How hash indexing enhances database performance 3. Real-world applications and examples Check it out here: https://lnkd.in/gPrrfpWD I would love to hear your thoughts and any experiences you've had with hash indexing. Let's discuss and share knowledge! #system_design #Database #MediumArticle #Database

Multi-Modal Indexing System in Apache Hudi. One of the core technical design choices that separates Apache Hudi from other open table formats is 𝐈𝐧𝐝𝐞𝐱𝐢𝐧𝐠. From its inception, Hudi has been heavily optimized to handle mutable change streams with varying write patterns. Indexing plays a critical role in dealing with these writes that deems faster updates. They help with faster upserts/deletes by quickly locating the records that needs to be updated. 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐇𝐮𝐝𝐢'𝐬 𝐌𝐮𝐥𝐭𝐢-𝐌𝐨𝐝𝐚𝐥 𝐢𝐧𝐝𝐞𝐱𝐢𝐧𝐠? Hudi's multi-modal indexing system redefines the design of an indexing subsystem for data lakes. 𝐇𝐨𝐰 𝐢𝐬 𝐢𝐭 𝐢𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐞𝐝? - By using Hudi's Metadata table - Hudi's metadata table is actually a single internal Hudi Merge-on-Read table - It consists of different types of indexes such as column stats, bloom filter, files, record-level, etc. as individual '𝘱𝘢𝘳𝘵𝘪𝘵𝘪𝘰𝘯𝘴' 𝐂𝐨𝐫𝐞 𝐃𝐞𝐬𝐢𝐠𝐧 𝐏𝐫𝐢𝐧𝐜𝐢𝐩𝐥𝐞𝐬 - Scalable metadata: Size of the metadata table can grow with larger updates. The table needs to scale metadata to TBs of size - ACID updates: index & table metadata must always be up-to-date & in sync with the data table. No partial writes - Fast Lookups: Indexes size can be huge. For certain queries to be efficient, scanning entire indexes should be avoided 𝐇𝐨𝐰 𝐝𝐨𝐞𝐬 𝐢𝐭 𝐢𝐦𝐩𝐫𝐨𝐯𝐞𝐬 𝐩𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞? ✅ File Listings: For large datasets, listing files is a huge bottleneck. With files index in a multi-modal index system, listing is drastically reduced Vs direct S3 listing ✅ Data Skipping: On the read-side, column_stats index (min, max val) can be used to skip irrelevant data files. This serves as a 2nd level pruning Vs reading each #Parquet footers directly ✅ Fast Upserts: Bloom_filter index stores the bloom filters of all data files, thereby avoiding the need to scan the footers of each data file. Record-level indexes helps locating records faster. Having worked on other open #lakehouse table formats, I understand specific pain points in terms of write latency and how a multi-modal indexing system can be extremely beneficial. Detailed read in comments. #dataengineering #softwareengineering

📖 Your Weekend Read: Have you checked out the latest #ScyllaDB Open Source 5.4.0 release? Tzach Livyatan, VP of Product, shares the new updates in his latest blog: https://ow.ly/xiLG50Qlpl9 #opensource #NoSQL #database

Here's the 2nd episode in my series focused on vector databases and vector search, this time with Linpeng Tang, the CTO of MyScale. There's lots going on in the vector world but what makes MyScale really interesting and unique is that they're combining the best of OLAP with vector search, bringing the world of structured and unstructured together. Also, because they started as a fork of ClickHouse, they're also able to scale beyond where many existing vector databases fail today. Enjoy! #vectorsearch #vectordatabase #llm #rag #clickhouse #analytics

In a new Vector Databases talk, our CTO Linpeng Tang shares our unique blend of SQL with Vector Search. We've come a long way since starting in 2019, turning ClickHouse into a scalable vector database. This step forward has opened new doors in data analytics, letting us find deeper insights from unstructured data easily. MyScale is working on new ideas that are making it easier to improve data analytics in the future. Discover how MyScale is changing the game in making data work smarter. #MyScale #Innovation #FutureOfData

Hello. Today is a day for some special stuff: 𝗛𝗼𝘄 𝘁𝗼 𝗯𝗶𝘁 𝘁𝗵𝗲 𝗱𝗮𝘁𝗮𝗯𝗮𝘀𝗲 𝗲𝗻𝗴𝗶𝗻𝗲 𝗳𝗿𝗼𝗺 𝘀𝗰𝗿𝗮𝘁𝗰𝗵. We have been using #AnchorModeling for mutable data from the very beginning. And it is persuaded that merge or colocated joins is the magic that makes it work. And even more — that there is nothing better for such a highly normalized data modeling approach. But we have used #Vertica and now #Trino for a very long time to know the limits. We know exactly that general-case implementation for joins in distributed analytical engines like Vertica has obvious limits when used with such atomized data structures. In the real world, to query highly normalized data, you have to join tens of tables. And you may assume that the larger the number of joins, the slower your query gets. It is not an irresistible rule in all possible cases because of cost-based optimizations. But even if all of the query tables are sorted by exactly the same field in the same manner, you can easily check that query duration increases dramatically while adding tens of new attributes to join. But why? Is a merge or colocated join magic fake? It does not. It is just not implemented properly. With anchor modeling, your data is extremely well organized for such algorithms, but it does not fit a general case implementation in your database engine. You can easily overperform it (in terms of time or resource consumption) by implementing the join technique from scratch with your own hands. And here is what we get in terms of time. Have a nice day, and never stop testing the limits.

Curious about how data works its magic? Join Serxan Hamzayev on JavatoDev Medium for a clear comparison between Elasticsearch and traditional databases. Whether you're a developer or a data enthusiast, this article breaks down the differences, offering insights for efficient data handling. Let's explore the world of data together! 🚀 #Elasticsearch #DataComparison #JavatoDev #TechInsights #DataHandling https://lnkd.in/gRCSWjTe