Learning a new database paradigm can feel like trying to make your way around a new town: there are a few sites that you recognize, but you have no sense of how they connect. ๐งย โฌ๏ธ If youโre new to Fluree or still trying to better understand Flureeโs data model, you may need a โmapโ to connect the virtual dots. This page from our docs book is a great tool for understanding the ways you can interact with Fluree. You'll learn: โข What it means for Fluree to be aย JSON-LD database,ย which is a version of anย RDF (Resource Description Framework) graph database โข What RDF is and how its data model compare to the relational model โข Which concepts you can apply from the relational world and which ones don't apply โข The relationship between this internal model of how data is represented and Fluree's interfaces for transacting and querying Start learning here: https://lnkd.in/eskXXyJQ #jsonld #relationaldatabases #rdf #datamodel
Flureeโs Post
More Relevant Posts
-
๐ Passionate Software Engineer | Azure, Python, Java & Golang Developer | โ๏ธ AZ-900ยฎ | ๐ DP-900ยฎ | Solving Data Integration Challenges | Offering Scalable Data Solutions
๐ ๐๐๐ญ๐ ๐๐๐ค๐๐ก๐จ๐ฎ๐ฌ๐: ๐๐ฉ๐๐๐ก๐ ๐๐๐๐๐๐ซ๐ ๐ โ ๐๐ก๐๐ญ ๐๐ฑ๐๐๐ญ๐ฅ๐ฒ ๐ข๐ฌ ๐๐ฉ๐๐๐ก๐ ๐๐๐๐๐๐ซ๐ ? ๐บ It is a table format created in 2017 by Netflixโs Ryan Blue and Daniel Weeks, for massive analytic datasets. ๐บ It overcame challenges with performance, consistency, and many of the challenges previously stated with the Hive table format. ๐บ Became open source in 2018. โ ๐๐ก๐๐ญ ๐๐ซ๐ ๐ข๐ญ๐ฌ ๐ค๐๐ฒ ๐๐๐๐ฎ๐ซ๐๐ฌ? ๐บ ๐บ๐๐๐๐๐ ๐ฌ๐๐๐๐๐๐๐๐ ๐๐๐ ๐ฝ๐๐๐๐๐๐๐๐๐ Iceberg supports schema evolution, allowing changes like adding, dropping, renaming columns, and updating column types without affecting query results or data consistency. It also provides versioning, enabling rollback to previous states. ๐บ ๐ท๐๐๐๐๐๐๐๐๐๐๐ Iceberg offers hidden partitioning that abstracts the complexity, optimizing query performance by automatically selecting the most efficient partitioning strategy based on the query workload. ๐บ ๐จ๐๐๐๐๐๐๐๐ ๐๐๐ ๐ช๐๐๐๐๐๐๐๐๐๐ Iceberg guarantees atomic operations and consistent reads through its design, which includes atomic commit protocols. This ensures that updates are all-or-nothing, preventing partial writes and maintaining data integrity. ๐บ ๐ซ๐๐๐ ๐ณ๐๐๐๐๐ ๐๐๐ ๐ฐ๐๐ ๐๐๐๐๐ Iceberg optimizes data layout and includes built-in indexing mechanisms, such as manifest files and metadata trees, which enhance query performance by pruning unnecessary data reads. ๐บ ๐ป๐๐๐ ๐ป๐๐๐๐๐ Iceberg supports time travel, allowing users to query data as it existed at any point in time. This facilitates easy analysis of historical data and recovery from accidental changes. ๐บ ๐ฐ๐๐๐๐๐๐๐๐๐๐๐๐๐๐๐ Iceberg is designed to be compatible with multiple processing engines such as Apache Spark, Apache Flink, and Trino, making it versatile and easy to integrate into existing data infrastructure. โ ๐๐๐ฅ๐ข๐๐๐ข๐ฅ๐ข๐ญ๐ฒ & ๐๐๐ซ๐๐จ๐ซ๐ฆ๐๐ง๐๐ Iceberg is designed to handle massive tables, often containing tens of petabytes of data by: ๐บ ๐ฌ๐๐๐๐๐๐๐๐ ๐บ๐๐๐ ๐ท๐๐๐๐๐๐๐: Iceberg enables rapid scan planning, eliminating the need for a distributed SQL engine to read tables or locate files. ๐บ ๐จ๐ ๐๐๐๐๐๐ ๐ญ๐๐๐๐๐๐๐๐: It optimizes data reading by pruning data files using partition and column-level statistics, leveraging table metadata to filter out unnecessary data. #dataengineering #ุชููุณ_ุฃูุถู
To view or add a comment, sign in
-
-
Excited to share my latest Medium blog post on LSM Trees! In "LSM Trees -A deep dive into a fundamental data structure used in databases," I explore the inner workings of LSM Trees. From write amplification to compaction, we'll cover it all. Whether you're a developer, database enthusiast, or just curious about tech, head on and dive in! Any suggestions are appreciated. Happy coding! #LSMTrees #Databases #MediumPost
LSM Trees -A deep dive into a fundamental data structure used in databases
medium.com
To view or add a comment, sign in
-
Staff Data Engineering Advocate @Onehouse.ai | Apache Hudi, Iceberg Contributor | Distributed Systems | Technical Author
Multi-Modal Indexing System in Apache Hudi. One of the core technical design choices that separates Apache Hudi from other open table formats is Indexing. From its inception, Hudi has been heavily optimized to handle mutable change streams with varying write patterns. Indexing plays a critical role in dealing with these writes that deems faster updates. They help with faster upserts/deletes by quickly locating the records that needs to be updated. ๐๐ก๐๐ญ ๐ข๐ฌ ๐๐ฎ๐๐ข'๐ฌ ๐๐ฎ๐ฅ๐ญ๐ข-๐๐จ๐๐๐ฅ ๐ข๐ง๐๐๐ฑ๐ข๐ง๐ ? Hudi's multi-modal indexing system redefines the design of an indexing subsystem for data lakes. ๐๐จ๐ฐ ๐ข๐ฌ ๐ข๐ญ ๐ข๐ฆ๐ฉ๐ฅ๐๐ฆ๐๐ง๐ญ๐๐? - By using Hudi's Metadata table - Hudi's metadata table is actually a single internal Hudi Merge-on-Read table - It consists of different types of indexes such as column stats, bloom filter, files, record-level, etc. as individual '๐ฑ๐ข๐ณ๐ต๐ช๐ต๐ช๐ฐ๐ฏ๐ด' ๐๐จ๐ซ๐ ๐๐๐ฌ๐ข๐ ๐ง ๐๐ซ๐ข๐ง๐๐ข๐ฉ๐ฅ๐๐ฌ - Scalable metadata: Size of the metadata table can grow with larger updates. The table needs to scale metadata to TBs of size - ACID updates: index & table metadata must always be up-to-date & in sync with the data table. No partial writes - Fast Lookups: Indexes size can be huge. For certain queries to be efficient, scanning entire indexes should be avoided ๐๐จ๐ฐ ๐๐จ๐๐ฌ ๐ข๐ญ ๐ข๐ฆ๐ฉ๐ซ๐จ๐ฏ๐๐ฌ ๐ฉ๐๐ซ๐๐จ๐ซ๐ฆ๐๐ง๐๐? โ File Listings: For large datasets, listing files is a huge bottleneck. With files index in a multi-modal index system, listing is drastically reduced Vs direct S3 listing โ Data Skipping: On the read-side, column_stats index (min, max val) can be used to skip irrelevant data files. This serves as a 2nd level pruning Vs reading each #Parquet footers directly โ Fast Upserts: Bloom_filter index stores the bloom filters of all data files, thereby avoiding the need to scan the footers of each data file. Record-level indexes helps locating records faster. Having worked on other open #lakehouse table formats, I understand specific pain points in terms of write latency and how a multi-modal indexing system can be extremely beneficial. Detailed read in comments. #dataengineering #softwareengineering
To view or add a comment, sign in
-
-
On Notice Period | Software Engineer @ Tekion Corp. | Data Science Professional having experience in Research and kaggle
๐ Unlocking Database Performance with Hash Indexing! ๐ As data grows exponentially, optimizing database performance becomes crucial for efficient data retrieval and management. I'm excited to share my latest article on Medium where I dive into the world of data structures, focusing on the powerful concept of Hash Indexing. ๐In this article, you'll discover: 1. The basics and importance of hash indexing 2. How hash indexing enhances database performance 3. Real-world applications and examples Check it out here: https://lnkd.in/gPrrfpWD I would love to hear your thoughts and any experiences you've had with hash indexing. Let's discuss and share knowledge! #system_design #Database #MediumArticle #Database
Essential Data Structures for Database Performance: Hash Indexing
medium.com
To view or add a comment, sign in
-
Staff Data Engineering Advocate @Onehouse.ai | Apache Hudi, Iceberg Contributor | Distributed Systems | Technical Author
Multi-Modal Indexing System in Apache Hudi. One of the core technical design choices that separates Apache Hudi from other open table formats is ๐๐ง๐๐๐ฑ๐ข๐ง๐ . From its inception, Hudi has been heavily optimized to handle mutable change streams with varying write patterns. Indexing plays a critical role in dealing with these writes that deems faster updates. They help with faster upserts/deletes by quickly locating the records that needs to be updated. ๐๐ก๐๐ญ ๐ข๐ฌ ๐๐ฎ๐๐ข'๐ฌ ๐๐ฎ๐ฅ๐ญ๐ข-๐๐จ๐๐๐ฅ ๐ข๐ง๐๐๐ฑ๐ข๐ง๐ ? Hudi's multi-modal indexing system redefines the design of an indexing subsystem for data lakes. ๐๐จ๐ฐ ๐ข๐ฌ ๐ข๐ญ ๐ข๐ฆ๐ฉ๐ฅ๐๐ฆ๐๐ง๐ญ๐๐? - By using Hudi's Metadata table - Hudi's metadata table is actually a single internal Hudi Merge-on-Read table - It consists of different types of indexes such as column stats, bloom filter, files, record-level, etc. as individual '๐ฑ๐ข๐ณ๐ต๐ช๐ต๐ช๐ฐ๐ฏ๐ด' ๐๐จ๐ซ๐ ๐๐๐ฌ๐ข๐ ๐ง ๐๐ซ๐ข๐ง๐๐ข๐ฉ๐ฅ๐๐ฌ - Scalable metadata: Size of the metadata table can grow with larger updates. The table needs to scale metadata to TBs of size - ACID updates: index & table metadata must always be up-to-date & in sync with the data table. No partial writes - Fast Lookups: Indexes size can be huge. For certain queries to be efficient, scanning entire indexes should be avoided ๐๐จ๐ฐ ๐๐จ๐๐ฌ ๐ข๐ญ ๐ข๐ฆ๐ฉ๐ซ๐จ๐ฏ๐๐ฌ ๐ฉ๐๐ซ๐๐จ๐ซ๐ฆ๐๐ง๐๐? โ File Listings: For large datasets, listing files is a huge bottleneck. With files index in a multi-modal index system, listing is drastically reduced Vs direct S3 listing โ Data Skipping: On the read-side, column_stats index (min, max val) can be used to skip irrelevant data files. This serves as a 2nd level pruning Vs reading each #Parquet footers directly โ Fast Upserts: Bloom_filter index stores the bloom filters of all data files, thereby avoiding the need to scan the footers of each data file. Record-level indexes helps locating records faster. Having worked on other open #lakehouse table formats, I understand specific pain points in terms of write latency and how a multi-modal indexing system can be extremely beneficial. Detailed read in comments. #dataengineering #softwareengineering
To view or add a comment, sign in
-
-
๐ Your Weekend Read: Have you checked out the latest #ScyllaDB Open Source 5.4.0 release? Tzach Livyatan, VP of Product, shares the new updates in his latest blog: https://ow.ly/xiLG50Qlpl9 #opensource #NoSQL #database
ScyllaDB Open Source 5.4: Wasm Functions & Aggregates, Strongly Consistent Topology Updates & More - ScyllaDB
scylladb.com
To view or add a comment, sign in
-
Head of Marketing & Developer Relations @ Skyflow | ex-Google | Podcast Host for Partially Redacted and Software Engineering Daily | โ๏ธ Snowflake Data Superhero | AWS Community Builder
Here's the 2nd episode in my series focused on vector databases and vector search, this time with Linpeng Tang, the CTO of MyScale. There's lots going on in the vector world but what makes MyScale really interesting and unique is that they're combining the best of OLAP with vector search, bringing the world of structured and unstructured together. Also, because they started as a fork of ClickHouse, they're also able to scale beyond where many existing vector databases fail today. Enjoy! #vectorsearch #vectordatabase #llm #rag #clickhouse #analytics
We're back with another episode in our series of Vector Databases. SQL Meets Vector Search with Linpeng Tang of MyScale ๐ In this episode we have with us Linpeng Tang, CTO and co-founder of MyScale. MyScale is a super interesting technology. They're combining the best of OLAP databases with Vector Search. The project started back in 2019 where they forked ClickHouse and then adapted it to support Vector Storage, Indexing, and Search. The really unique and cool thing is you get the familiarity and usability of SQL with the power of being able to compare the similarity between unstructured data. We think this has really fascinating use cases for analytics well beyond what we're seeing with other vector database technology that's mostly restricted to building RAG models for LLMs. Also, because it's built on ClickHouse, MyScale is massively scalable, which is an area that many of the dedicated vector databases actually struggle with. We cover a lot about how vector databases work, why they decided to build off of ClickHouse, and how they plan to open source the database. Watch On Youtube: https://lnkd.in/gGFiwVND Timestamps Introduction [02:29] Value of a Vector Database [06:22] Forking ClickHouse [12:40] Transforming Clickhouse into a SQL vector database [18:53] Data modeling [32:08] What data can be Vectorized [32:56] Indexing [38:37] Achieving Scale [43:35] Bottlenecks [46:35] MyScale vs other dedicated Vector Databases [48:41] Going Open Source [51:38] Closing thoughts [56:04] Linpeng Tang | Sean Falconer | Alex DeBrie
To view or add a comment, sign in
-
LLMs and Python Developer | Generative AI, LangChain, Fast API | I help people design and create AI applications
In a new Vector Databases talk, our CTO Linpeng Tang shares our unique blend of SQL with Vector Search. We've come a long way since starting in 2019, turning ClickHouse into a scalable vector database. This step forward has opened new doors in data analytics, letting us find deeper insights from unstructured data easily. MyScale is working on new ideas that are making it easier to improve data analytics in the future. Discover how MyScale is changing the game in making data work smarter. #MyScale #Innovation #FutureOfData
We're back with another episode in our series of Vector Databases. SQL Meets Vector Search with Linpeng Tang of MyScale ๐ In this episode we have with us Linpeng Tang, CTO and co-founder of MyScale. MyScale is a super interesting technology. They're combining the best of OLAP databases with Vector Search. The project started back in 2019 where they forked ClickHouse and then adapted it to support Vector Storage, Indexing, and Search. The really unique and cool thing is you get the familiarity and usability of SQL with the power of being able to compare the similarity between unstructured data. We think this has really fascinating use cases for analytics well beyond what we're seeing with other vector database technology that's mostly restricted to building RAG models for LLMs. Also, because it's built on ClickHouse, MyScale is massively scalable, which is an area that many of the dedicated vector databases actually struggle with. We cover a lot about how vector databases work, why they decided to build off of ClickHouse, and how they plan to open source the database. Watch On Youtube: https://lnkd.in/gGFiwVND Timestamps Introduction [02:29] Value of a Vector Database [06:22] Forking ClickHouse [12:40] Transforming Clickhouse into a SQL vector database [18:53] Data modeling [32:08] What data can be Vectorized [32:56] Indexing [38:37] Achieving Scale [43:35] Bottlenecks [46:35] MyScale vs other dedicated Vector Databases [48:41] Going Open Source [51:38] Closing thoughts [56:04] Linpeng Tang | Sean Falconer | Alex DeBrie
To view or add a comment, sign in
-
Hello. Today is a day for some special stuff: ๐๐ผ๐ ๐๐ผ ๐ฏ๐ถ๐ ๐๐ต๐ฒ ๐ฑ๐ฎ๐๐ฎ๐ฏ๐ฎ๐๐ฒ ๐ฒ๐ป๐ด๐ถ๐ป๐ฒ ๐ณ๐ฟ๐ผ๐บ ๐๐ฐ๐ฟ๐ฎ๐๐ฐ๐ต. We have been using #AnchorModeling for mutable data from the very beginning. And it is persuaded that merge or colocated joins is the magic that makes it work. And even more โ that there is nothing better for such a highly normalized data modeling approach. But we have used #Vertica and now #Trino for a very long time to know the limits. We know exactly that general-case implementation for joins in distributed analytical engines like Vertica has obvious limits when used with such atomized data structures. In the real world, to query highly normalized data, you have to join tens of tables. And you may assume that the larger the number of joins, the slower your query gets. It is not an irresistible rule in all possible cases because of cost-based optimizations. But even if all of the query tables are sorted by exactly the same field in the same manner, you can easily check that query duration increases dramatically while adding tens of new attributes to join. But why? Is a merge or colocated join magic fake? It does not. It is just not implemented properly. With anchor modeling, your data is extremely well organized for such algorithms, but it does not fit a general case implementation in your database engine. You can easily overperform it (in terms of time or resource consumption) by implementing the join technique from scratch with your own hands. And here is what we get in terms of time. Have a nice day, and never stop testing the limits.
To view or add a comment, sign in
-
-
Curious about how data works its magic? Join Serxan Hamzayev on JavatoDev Medium for a clear comparison between Elasticsearch and traditional databases. Whether you're a developer or a data enthusiast, this article breaks down the differences, offering insights for efficient data handling. Let's explore the world of data together! ๐ #Elasticsearch #DataComparison #JavatoDev #TechInsights #DataHandling https://lnkd.in/gRCSWjTe
Unlocking Data Magic: Elasticsearch vs. The Database World
medium.com
To view or add a comment, sign in