12/14/2023 0 Comments Airflow kubernetes installThe Chart.yaml file uses the Airflow as dependency and overwrites its default parameters in the values.yaml file: airflow:Īs an executor, we are using the KubernetesExecutor which allows us to spawn our tasks across different worker groups in our Kubernetes cluster with the natively supported scaling capability of Kubernetes. The big benefit of this approach is that values of the official helm chart can be easily updated and reduce the complexity of the high number of input parameters: # helm/Chart.yaml For that, I created an own helm chart that inherits the official Helm chart. However, there is now also a helm/ folder that contains the Airflow package for our Kubernetes Cluster. The folder structure is similar to the original structure with the dags/ and plugins/ folder. I will explain it later step by step: my-airflow/ In the following, we will use the following folder structure. Helm 3 as a package manager for Kubernetesįor this tutorial, we are using the official Airflow Helm chart to set Airflow up.Requirementsįor this tutorial, I assume that you are familiar with: Within this tutorial, I show how you can setup your Airflow within a Kubernetes Cluster with the particular focus on a local development setup. The newer Kubernetes Executor can solve this issue because of the possibilities that Kubernetes provides you natively. However, this “traditional” executor do have the scaling limitations like only a single worker type as well as scaling only depending on the number of tasks running on a cluster instead of the actual worker load of the dedicated tasks. For the Celery Executor, which is the most popular one if you are running Airflow in a containerised or “bare-metal” environment. The developers can model their data transformation steps as a Directed Acyclic Graph (DAG), which is written in Python. Especially for data warehouses and its data transformation processes, Airflow helps to schedule the tasks in the defined order. Apache Airflow is currently one of the most popular task orchestration tools available.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |