Ohad Levi’s Post

Week #6 of building a #storage service Past few weeks i discussed about object persistence, replication and fault tolerance. So, before I investigate the nuances and technical difficulties that each tier in the storage service presents, the plan is to roll out a completely working MVP. This week's focus is search. I was looking out for products that provide search as a feature on the stored objects. There were a bunch of products which had family integrations like s3 with elastic search. Similarly with other service providers like #gcp and #azure. The goal here shifts slightly, our object storage needs to support integration with these external services now. Putting aside my slight bias for a unified theory where everything is built into the product, I'm taking some time to explore a couple of services and how they can be integrated with our Object store. #Meilisearch - Meili search is a rust based search offering. Like always here is my #kta aka key take aways from this exploration. 1. You can easily load your JSON documents onto meili search and it gives you out of the box semantic search with a nice UI. 2. When you load your documents, Meili Search refers to this process as creating an index. For instance, if you have a million JSON docs related to movies, you can create an index for them in a matter of seconds. 3. Meili creates multiple in memory/persistent data structures: i. Inverted Indexes: These maps each word in a document to a set of documents. ii. Finite State Transducers: Utilized for string mapping operations. iii. Rtrees: These are used for effective spatial and multi-dimensional search capabilities, enhancing search efficiency for location based data. iv. Query Graphs: Meili Search generates all possible variations of user queries along with possible typo's. 4. Few Notable Algorithms: i. Levenshtein Algorithm - For edit distance calculations. ii. Bucket sort - Sorts documents based on some ranking rules Here's the link to the deep dive of their engine. https://lnkd.in/gU9DgesC Discoveries for future learning http://www.lmdb.tech/doc/ #computerscience #storage #database #algorithms #datastructures #product #software #softwaredevelopment

Vector search on object storage is stimulating new thinking about database design. I like this article from turbopuffer a lot. There's a creative mix of economic and engineering design to reduce cost while preserving performance. Food for thought! https://lnkd.in/gHUGtZM9 #s3 #vectorsearch #opensearch #bigdata #elasticsearch

The word "vector database" sparks excitement in tech community. But how do vector databases work, and what are they useful for? I wrote about it in my new post https://lnkd.in/gwSYr25i

Master These 8 Data Structures and Conquer Any Tech Challenge Ready to level up your coding game and tackle complex data problems like a pro? Buckle up, because we're diving deep into 8 essential data structures and their real-world impact: B-Tree: Your go-to for organising and searching massive datasets like databases and file systems. Think of it as a super-efficient filing cabinet that lets you find and insert data in a flash! Hash Table: The champion of lightning-fast data retrieval through key-value pairs. This marvel is used in caching, symbol tables, and databases, offering instant lookups for your data needs. Trie: Master of handling dynamic data and representing hierarchical structures like dictionaries.This beauty shines in autocomplete features and prefix-based searches, making it a star for dynamic data handling. Bloom Filter: The space-saving detective of the data world, this clever structure tells you if an element belongs to a set with minimal memory. Spell check, caching, and network routers rely on its efficient membershio testina. Inverted Index: The secret weapon of search engines, this mastermind maps words to documents, enabling blazing-fast full-text searches. No wonder it's the backbone of information retrieval systems! Skip List: The versatile champion of fast searching, insertion, and deletion. This hybrid structure shines in in-memory databases and priority queues, balancing simplicity and efficiency like a boss. Log-Structured Merge (LSM) Tree: The write-intensive workload warrior, this tree optimizes data storage with its high write throughput and efficient compaction. Think of it as the ultimate data storage powerhouse for key-value stores and distributed databases. SSTable (Sorted String Table): The persistent storage superhero, this structure stores large datasets with blazing-fast reads and writes. It's the go-to for systems like Apache Cassandri, offering a sorted, immutable table structure for rock-solid performance in distributed environments. Mastering these data structures is your key to building robust, scalable systems. So keep exploring, innovating, and conquering those tech challenges! #BTree #binarytree #hashing #hashtable #bloomfilter #stable #invertedindex #skiplist #Ismtree #trie #datastructure #usecases

Here are some of the most commonly used indexing algorithms

Handling vast data can lead to cumbersome and slow workflows. How can you streamline your data-intensive projects? 🔬 One of our clients, a research center, needed a workflow to streamline genome data collection, storage, and retrieval. We are committed to bringing tailored solutions to our clients. Along with them, we decided to run a proof-of-concept workflow with Elasticsearch as the core data storage and indexing piece, working hand in hand and adapting to their unique needs. This is what the researchers achieved. Does any of the following resonate with you? 1- 🫳Zero errors —> An intuitive and straightforward web-based interface allowed them to avoid errors while entering parameters. 2- 🗃️ Minimizing time spent on file management —> They don't have to waste time moving and organizing files. 3- 🔎 Focus on what they do best: research —> After bringing quick, intuitive searches in non-relational databases, researchers could concentrate on their core capabilities. Discover the full story of this ground-breaking project on our kommit's blog. Dive in here 👇 https://bit.ly/3T1BHEd #Elasticsearch #TechSolutions #innovation

Master These 8 Data Structures and Conquer Any Tech Challenge Ready to level up your coding game and tackle complex data problems like a pro? Buckle up, because we're diving deep into 8 essential data structures and their real-world impact: B-Tree: Your go-to for organising and searching massive datasets like databases and file systems. Think of it as a super-efficient filing cabinet that lets you find and insert data in a flash! Hash Table: The champion of lightning-fast data retrieval through key-value pairs. This marvel is used in caching, symbol tables, and databases, offering instant lookups for your data needs. Trie: Master of handling dynamic data and representing hierarchical structures like dictionaries. This beauty shines in autocomplete features and prefix-based searches, making it a star for dynamic data handling. Bloom Filter: The space-saving detective of the data world, this clever structure tells you if an element belongs to a set with minimal memory. Spell check, caching, and network routers rely on its efficient membership testing. Inverted Index: The secret weapon of search engines, this mastermind maps words to documents, enabling blazing-fast full-text searches. No wonder it's the backbone of information retrieval systems! Skip List: The versatile champion of fast searching, insertion, and deletion. This hybrid structure shines in in-memory databases and priority queues, balancing simplicity and efficiency like a boss. Log-Structured Merge (LSM) Tree: The write-intensive workload warrior, this tree optimizes data storage with its high write throughput and efficient compaction. Think of it as the ultimate data storage powerhouse for key-value stores and distributed databases. SSTable (Sorted String Table): The persistent storage superhero, this structure stores large datasets with blazing-fast reads and writes. It's the go-to for systems like Apache Cassandra, offering a sorted, immutable table structure for rock-solid performance in distributed environments. Mastering these data structures is your key to building robust, scalable systems. So keep exploring, innovating, and conquering those tech challenges! Follow Aastha Gupta for more content :) #BTree #binarytree #hashing #hashtable #bloomfilter #sstable #invertedindex #skiplist #lsmtree #trie #datastructure #usecases

Week #10 of building a storage service. Last week we discussed a case for client protocol redesign where i spoke about how network serialization formats heavily influences I/O. This week I looked into LanceDB serialization mechanism for persistent storage aka how is data actually being stored on disk. Lance recently introduced a really cool technique for the developers to take control over their encoding mechamism by making use of protobufs. Which means as a dev you just have to define your *.proto file and your readers and writers will make use of that to understand how they encode and decode the files. This let's the community figure out new techniques for encoding/decoding data and publish the .proto file for others to use it, with no changes to the system and optimise for their workloads. This essentially overcomes some of parquet's encoding workflows. Encodings - Parquet. https://lnkd.in/gs5_kTxB Here's my #kta aka key take aways from lance's new storage ideology. 1. Lance does not define how your data is serialized into in your file but the developer extensions do. 2. Configuring the optimal size for in memory data structures and persistent data structures have always been a pain in the a** mostly because of row groups, hence they moved to continous column writing and reading where you can configure your page sizes accordingly. Note this is just related serialization and deserialization you can checkout other improvements and benefits here. https://lnkd.in/gd4WGSHP #storage #databases #computerscience #filesystems #encodings

Splink 3.9.11 (just released) is >10x faster on high core count machines with DuckDB.  DuckDB is now recommended even for very large data linkages/dedupes of 10s of millions of records +. Benchmarking results demonstrate deduping a 7m record dataset with 1bn comparisons is possible in just 2 minutes - making Splink (AFAIK!) by far the fastest free data deduplication library. See detailed results here: https://lnkd.in/eHCYFEqr I'm super pleased with these results because they dramatically lower the barriers to entry in using Splink. No more need to setting up a Spark cluster, fiddling with complex Spark config and registering UDFs. Just pip install splink and use the DuckDB backend. Discuss on HN here: https://lnkd.in/e92ZARkz

📐 Vector Search Optimization via KMeans, Voronoi Cells and Inverted File Index (aka “Cell-Probing”). I had quite a lot of fun 🤩 understanding how to improve vector search without impacting OLTP performances. 😯 As usual, a divide-and-conquer plus low-friction provisioning offered by containers to spin up a compute node was the solution. 🚀 Here's what I did, so that you can do it too, no matter which version of #SQLServer or #AzureSQL you are using. It is fun, you learn a lot and it is practical. The article also shows quite clearly all the reasons why *I love* 😍 being a developer: lateral thinking, creativity, and engineering all together. Fantastic! 💪