Apache Airflow’s Post

Thrilled to share an insightful blog post by Community Member Harshal Pagar on Running Databricks Notebook with Airflow. This article delves into the integration of Databricks Notebooks with Apache Airflow, offering a step-by-step guide on how to leverage this powerful combination for your data workflows. Whether you're looking to streamline your data processing or explore new capabilities, this post is a must-read! Kudos to the author for providing such valuable content. Let’s dive in, learn, and continue to innovate together! #ApacheAirflow #Databricks #DataEngineering #DataScience #BigData #DataPipeline

🚀 👋 Hello LinkedIn community!" I'm excited to share key points from yesterday's post, exploring the details of data pipelines and how Pyspark is implemented in Databricks.". 🌐✨ 📽️ Video Talk Highlights: 🔗 Title: "Analyzing Logs with Azure Event Hub in Real-Time" 🔧 Tools: Databricks Spark, PySpark 🔍 What's Covered: - Setting up Databricks Spark on a personal compute environment (not shared cluster) – for more information, check Compute Access Mode Limitations on Azure Databricks. - Using PySpark to read data streams from Azure Event Hub. - Real-time transformation of log data. 📊 In the era of big data, real-time analytics is crucial. 🚀 Let's Discuss! Did you catch yesterday's post on data pipelines and Spark implementation? Share your thoughts and join the conversation in the comments! #Databricks #Spark #DataPipelines #PySpark #DataEngineering 

Recently, I encountered an issue while running a table that was failing on one Databricks cluster but working perfectly on another, and I discovered that the root cause was garbage collection. 🚀 If you’ve ever dealt with large-scale data processing in Databricks, you might have faced performance slowdowns due to garbage collection (GC). Here are some strategies to optimize and overcome these challenges: 1. Optimize Cluster Configuration: • Ensure your cluster has sufficient memory. Tune the executor memory using spark.executor.memory to prevent frequent GC events. 2. Tune Garbage Collection: • Experiment with GC algorithms like G1GC (-XX:+UseG1GC) for large heaps. Adjust JVM options to control GC behavior and reduce pause times. 3. Optimize Your Spark Code: • Minimize large shuffle operations and only persist necessary datasets to reduce memory usage and GC frequency. 4. Monitor and Diagnose: • Enable GC logging and leverage Databricks Ganglia to monitor memory usage and GC activity for better insights. 5. Scale Your Cluster: • Utilize Databricks auto-scaling to handle memory pressure dynamically and distribute workloads across more executors. 6. Upgrade Databricks Runtime: • Upgrade to the latest runtime version to take advantage of improved memory management and GC optimizations. Implementing these tips can lead to more stable and efficient processing in your Databricks clusters. Have you encountered GC issues? How did you tackle them? #DataEngineering #BigData #Databricks #Spark #CloudComputing #PerformanceOptimization

🚀 What Can We Do with Databricks? 🚀 Databricks is a unified analytics platform built on Apache Spark, designed to empower Data Engineers, Analysts, and Scientists. Whether you're building data pipelines, analyzing data, or creating machine learning models, Databricks offers the tools and scalability you need to succeed. Here’s how Databricks can elevate your work: #DataEngineering #BigData #ETL #ApacheSpark #CloudComputing #DataAnalytics #DataVisualization #BusinessIntelligence #DataDriven #TechInnovation #DataScience #MachineLearning #PredictiveAnalytics #MLPipelines #ApacheSpark

There is a misconception that to use databricks and later move to open source set up, it is just 'lift and shift', it is wrong. Open table format: Delta lake is quite free. It lives as a OSS and have its own ecosystem, the same with Apache Spark. The problem starts when using proprietary pieces of databricks offering. Such as autoloader, workflows, unity catalog, and more. Depends of the nature of the company this movement can make your data offering weaker. Specially, when looking from a governance pattern. Unity Catalog itself have pretty poor governance compared to Snowflake RBAC or bigquery. The Spark optimizations that comes with databricks won't be easy till replicate the same with orchestration workflows (open again the Airflow friend? or use Kubernetes argo workflow?). Again the overhead will have a cost. More now than ever offering such as Snowflake or Google Bigquery will be valuable to present business data because will make us to focus in the point. #dataanalytics #dataarchitecture

🚀 Unlocking Cost Savings with Databricks System Tables! 🚀 I'm thrilled to share how I leveraged Databricks system tables to analyze our usage costs and make informed decisions that led to a remarkable 50% reduction in our daily Databricks expenses! 🎉 But first, let me introduce you to this game-changing feature: ➔ What Are Databricks System Tables? Databricks system tables provide a wealth of information to help you monitor and optimize your Databricks environment: - Audit Logs: Keep track of activities at `system.access.audit` - Billable Usage Logs: Stay on top of costs with `system.billing.usage` - Pricing: Access list prices through `system.billing.list_prices` - Table & Column Lineage: Discover lineage information in `system.access` ➔ How They Work: - Unity Catalog: To get started, you need at least one Unity Catalog-enabled workspace. - Account Admins: System tables are enabled by account admins via the Unity Catalog REST API. - Permissions: Admins grant USE and SELECT permissions for users to access the data. ➔ Where To Find Them: - Catalog: System tables are available under the `system` catalog. - Schemas: The catalog includes `access` and `billing` schemas. Now, here's how I put these system tables to work: ➔ Cost Forecasting and Reduction: By joining the usage data with the `system.billing.list_prices` table, I gained valuable insights into our costs. This allowed me to identify areas for optimization and make informed decisions to reduce our expenses. ➔ Optimizing Our Setup: Armed with the insights from the system tables, I carefully analyzed our cluster and DLT configurations. By making strategic adjustments, I was able to significantly improve efficiency and cut down on unnecessary costs. The result? A staggering 50% reduction in our daily Databricks expenses!🙌 Databricks system tables offer a powerful toolset for understanding and optimizing your usage and costs. By leveraging this feature, you can make data-driven decisions that directly impact your bottom line. #Databricks #DataEngineering #CostReduction #BigData #CloudComputing #UnityCatalog #DataAnalytics

𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗶𝗻 𝗦𝗽𝗮𝗿𝗸 𝗮𝗻𝗱 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀: 𝗔 𝗦𝘁𝗼𝗿𝘆 𝗼𝗳 𝗨𝗻𝗹𝗲𝗮𝘀𝗵𝗶𝗻𝗴 𝘁𝗵𝗲 𝗙𝘂𝗹𝗹 𝗣𝗼𝘁𝗲𝗻𝘁𝗶𝗮𝗹 In the ever-evolving world of big data, Spark and Databricks have become indispensable tools for handling vast datasets and complex workloads. However, the true power of these platforms is only unlocked when their performance is finely tuned. I remember a time when I was tasked with optimizing a large-scale data pipeline on Databricks. The challenge was immense: the processing time was dragging on for hours, and costs were skyrocketing. It was clear that something had to change. I started by diving deep into cluster-level configurations, experimenting with various instance types and autoscaling settings. Then came the code-level optimizations—refining Spark jobs, tweaking memory management, and improving shuffles. But that wasn’t enough. I also had to rethink data management strategies, implementing partitioning and caching where needed. The results were transformative. Not only did we cut down the processing time by half, but we also significantly reduced operational costs. This experience reinforced the importance of a holistic approach to performance optimization—one that considers every layer, from infrastructure to code. In this guide, I’ll share the insights and strategies I’ve gathered along the way. Whether you're looking to improve efficiency, reduce costs, or simply make your Spark and Databricks workloads run smoother, these tips will help you get there. Stay tuned for more in-depth discussions on cluster configurations, code optimizations, and data management techniques! 🚀 #BigData #DataEngineering #Spark #Databricks #PerformanceOptimization #DataManagement #CloudComputing

Use #databricks delta.tuneFileSizesForRewrites table property. When this property is set to true, Databricks will automatically tune the file sizes based on workloads. For example, if you do a lot of merges on the Delta table, then the files will automatically be tuned to much smaller sizes than 1GB to accelerate the merge operation. Always explicitly broadcast smaller tables using hints or PySpark broadcast function Why do we need to explicitly broadcast smaller tables if AQE can automatically broadcast smaller tables for us? The reason for this is that AQE optimizes queries while they are being executed. Spark needs to shuffle the data on both sides and then only AQE can alter the physical plan based on the statistics of the shuffle stage and convert to broadcast join Therefore, if you explicitly broadcast smaller tables using hints, it skips the shuffle altogether and your job will not need to wait for AQE’s intervention to optimize the plan. Never broadcast a table bigger than 1GB because broadcast happens via the driver and a 1GB+ table will either cause OOM on the driver or make the drive unresponsive due to large GC pauses

🚀 Exciting News for #deltalake Users! 🚀 Say goodbye to inefficient storage utilization and performance bottlenecks! With deletion vectors in #deltalake in #databricks, you can now optimize storage and enhance performance for operations like updates and deletes. Unlike traditional methods, deletion vectors track soft deletes without rewriting entire Parquet files, ensuring swift DML latency. This revolutionary approach not only saves time but also preserves the integrity of your data, enabling seamless time travel while maximizing efficiency. Upgrade to Delta Lake today and experience the future of data management! #DeltaLake #DataManagement #StorageOptimization Databricks #PerformanceEnhancement #spark #databricks #data #DataIntegrity #DataQuality #TechSolution #azuredatabricks #azure #azuredataengineer #dataengineer #aws #databricks #amazons3 #azuredatalake #dataengineering #datafactory #cloudcomputing #microsoftazure #data #sparksql #spark #apache