Cluster Setup file(Filename: kind-local-cluster-setup.yaml)

# For local kind cluster setup
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "node=worker_1"
  extraMounts:
    - hostPath: ./data
      containerPath: /tmp/data
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "node=worker_2"
  extraMounts:
    - hostPath: ./data
      containerPath: /tmp/data 
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "node=worker_3"
  extraMounts:
    - hostPath: ./data
      containerPath: /tmp/data
            

Create cluster using below command:

You can do some checks

To deploy Airflow on Kubernetes, the first step is to create a namespace.

Then, thanks to Helm, you need to fetch the official Helm of Apache Airflow that will magically get deployed on your cluster. Or, almost magically 😅

After a few minutes, you should be able to see your Pods running, corresponding to the different Airflow components.

Don’t hesitate to check the current Helm release of your application with

Basically, each time your deploy a new version of your Airflow chart (after a modification or an update), you will obtain a new release. One of the most important field to take a look at is REVISION. This number will increase, if you made a mistake you can rollback to a previous revision with helm rollback.

To access the Airflow UI, open a new terminal and execute the following command

Listen on port 8080 on all addresses, forwarding to 8080

Make sure database settings present in variables.yaml (POSTGRES_HOST and POSTGRES_PORT ) and the ones present in values.yml are pointing to the sbaiv2 database

If you have some variables or connections that you want to export each time your Airflow instance gets deployed, you can define a ConfigMap. Open variables.yaml. This ConfigMap will export the environment variables under data. Great to have some bootstrap connections/variables.

Build the custom docker image and load it into your local Kubernetes cluster

Note : If you are using MAC M1 machine please use following command to build image :

Logs with Airflow on Kubernetes

With the KubernetesExecutor, when a task is triggered a POD is created. Once the task is completed, the corresponding POD gets deleted and so the logs. Therefore, you need to find a way to store your logs somewhere so that you can still access them. For local development, the easiest way is to configure a HostPath PV. Let’s do it! First thing first, you should already have a folder data/ created next to the file kind-custer.yaml.

Next, to provide a durable location to prevent data from being lost, you have to set up the Persistent Volume.

Then, you create a Persistent Volume Claim so that you bind the PV with Airflow.

And finally, deploy Airflow on Kubernetes again.

Some useful commands related to Helm(Rollback to specific deployment)

In case of timeout error please increase timeout by adding --timeout=15m.

The logs should be accessible in infra-setup/data folder. Please check that this folder has the right permissions. You can do so by executing the following command:

Next Steps

• If the installation is done on a VM, it's possible to create a network tunnel to be able to view the Django admin and Airflow UI on local machine by running the following command with the port numbers for Airflow or Django:

• The pipelines are setup to ingest data from files hosted on s3. If you're using this storage service, you need to make sure your machine has IAM role to access ,list , delete and update files on s3 bucket you’re using for dags. When working with VMs, please contact gamma platform for creating IAM role to enable access to the s3 bucket.

Debugging and useful commands

Here are some commands to help you debug errors.

• To list pods and check which one is failing :

• To check logs for specific pod :

• To execute scheduler pod and access airflow folders:

• To restore a specific version deployed

• To describe pod execution :

• In case of DAG Processor timeout error please increase timeout for the following parameters(AIRFLOW__CORE__DAG_FILE_PROCESSOR_TIMEOUT) in values.yaml file in pipeline/infra .Make sure to build the image and deploy it again.