Why are our new MF4 decoders awesome? 🔥
🔍 We recently added a new free software tool for the CANedge, which we call 'MF4 decoders'. These are simple executables that let you DBC decode your raw CAN/LIN data into CSV or Parquet files via drag & drop (or CLI).
Pretty basic, right?
🌫 However, what this does is to provide a simple, consistent and efficient method for creating Parquet data lakes - with zero coding required. This includes optimized folder structuring, partitioning, file concatenation and more - non-trivial stuff.
"OK, but what's so great about Parquet data lakes?"
Well, quite a lot - and more than I can explain in this post.
However, I did create a visual to illustrate one core feature: You can store your data lake anywhere - and integrate with everything.
☁️ Parquet data lakes are simply a bunch of folders containing binary files - exactly like our MF4 log files from the CANedge. This means you can store them locally, on S3, Azure blob storage, Google Drive etc. - and it makes storing that data up to 95% lower cost vs. databases.
⚙️ Further, Parquet files are directly supported in a wide array of tools such as #PowerBI, #Python, #MATLAB and more. Compared to MF4 files, Parquet files are probably >100x more widely supported.
🔗 But the true magic happens when you combine Parquet data lakes with various 'interfaces' such as Amazon #Athena, Google #BigQuery, Azure #Synapse, #DuckDB, #ClickHouse and more.
These tools essentially super charge your data lake queries, boosting performance and speed to extreme levels. Further, they expose your data lake via standard SQL, literally exploding the number of possible integrations.
We will deep dive on many of these integrations over the coming months - starting with #Grafana and #Excel. But I recommend to already now check out our article on the MF4 decoders below and to try them out on your own data - any feedback is encouraged!
https://lnkd.in/dj7xJ8us
#canbus #parquet #datalake #j1939 #telematics Amazon Web Services (AWS) CAN in Automation (CiA) Grafana Labs DuckDB ClickHouse MathWorks
Head of AI Capability at 10 Downing Street
7moDuckDB should get so much more love.