Apache Airflow’s Post

🚀 Exciting News from the Apache Airflow community! They've just released Apache Airflow 2.9.2 🎉 This update brings enhanced features and improvements, continuing to push the boundaries of data workflow automation. A huge thank you to all the contributors who made this release possible. Your hard work and dedication keep Apache Airflow moving forward! 🚀 #ApacheAirflow #Airflow #DataEngineering #OpenSource #Python #BigData #DataPipeline #WorkflowAutomation #DevOps #TechNews - The Data Engineerz Team

Apache Airflow 2.8.0 has just been released! It is the most popular software for orchestrating data workflows, and for good reasons. Unfortunately, you still need to be a programmer to utilize Airflow for creating your own workflows. 🙁 Enter Domino! With its intuitive graphical user interface, Domino empowers everyone to easily create and monitor their own data workflows, no programming required. 😎 Domino relies on Apache Airflow for orchestration, so you get the best of both worlds: a user-friendly interface coupled with a robust and reliable backend system. 💪 Try Domino today: https://lnkd.in/dGyHFc3m Github: https://lnkd.in/d6CfjwCB #domino #data #ai #machinelearning #workflows #apacheairflow #nocode #opensource #opensourcecommunity #workflowmanagement

Now you can use K8s native Horizontal Pod Autoscaler in Airflow Workers to support autoscaling based on external custom metrics as well as MySQL with KEDA #devops

Made my first contribution to Apache Datafusion-Comet. Datafusion-Comet tries to accelerate Spark by delegating certain operations to Apache Datafusion, an open-source query engine written in rust, part of the Apache Arrow ecosystem. https://lnkd.in/gGnsYB-x

Just finished the course “Learning Apache Airflow”! #apacheairflow #itautomation https://lnkd.in/eQgvzybF #apacheairflow #itautomation.

Just finished the course “Learning Apache Airflow” by Janani Ravi! Check it out: https://lnkd.in/gvzfYqa8 #apacheairflow #itautomation #technology #linkedin #Tech #DigitalTransformation #FutureOfWork

🖼️ Airflow 2.10.0 We rolled out a new version of Apache Airflow — 2.10.0. There are many useful changes and fixes in the release, as well as new features: ▶️ decorators @skip_if and @run_if with which you can conveniently set the conditions for starting or skipping a Task ▶️ Datasets are no longer running (https://lnkd.in/gC59KBQi ) inactive DAGs ▶️ Simultaneous use of multiple Executors (https://lnkd.in/gfr8_wgz ) within the framework of a single DAG ▶️ try_number is no longer increasing (https://lnkd.in/gfPDuwTZ ) during the execution of the Task ▶️ Important! Now Airflow collects telemetry by default (https://lnkd.in/gAfWEFPC ), all data is transferred to the Scarf (https://lnkd.in/gaUDj4TJ ). You can disable it, you need to fix the config [usage_data_collection]enabled=False or the environment variable SCARF_ANALYTICS=false See the full list of changes here (https://lnkd.in/gkDtsKcX ) or here (https://lnkd.in/g58CZ8Dt).

🚀 Getting Started with Apache Airflow! 🖥️ I'm excited to share my latest blog where I dive into the essentials of using Apache Airflow. Whether you're a data engineer, developer, or just curious about workflow orchestration, this guide will help you understand the fundamentals and get started quickly. Highlights:  ●  Overview of Apache Airflow: What it is and why it's a game-changer for workflow management.  ●  Key Concepts: Learn about DAGs, tasks, operators, sensors, XCom, and dependencies.  ●  Apache UI: Navigate the powerful Airflow UI to manage and monitor your workflows.  ●  Practical Examples: Hands-on example to help you create and manage your own workflows. #ApacheAirflow #WorkflowAutomation #DAGs #ETL #Python

Just finished the course “Learning Apache Airflow” by Janani Ravi! Check it out: https://lnkd.in/eMMqwNdY #apacheairflow #itautomation. It's a great course for getting started with Apache Airflow and learning some basics.

Apache XTable - What’s, Why’s and How’s? Apache Hudi, Apache Iceberg & Delta Lake provide a table-like abstraction on top of the native file formats like #Parquet. They serve as a metadata layer and offer necessary primitives for compute engines to interact with the storage. While these formats have enabled organizations to store data in an independent open tier, decoupled from compute, the decision to select and stick to one particular format is challenging. The question is — ‘Is there another way?’ I jotted all of it in the blog linked in comments. #dataengineering #softwareengineering