datapains-argo-cd-k8s

Workshops

Trino on K8S

This workshop is a continuation off: Deploy Trino on Kubernetes: Helm Setup with Hive Metastore & MinIO Integration

Medium Article Covering this repo.

Pre requisites

• docker-desktop with kubernetes enabled

Setup

Local

Set for your environment ~/.baschrc or ~/.zshrc or in the session where you run this.

export HELM_EXPERIMENTAL_OCI=1

helm repo add argo https://argoproj.github.io/argo-helm

List version

helm search repo argo-cd

Time of writing this using 7.3.4.

Core

make deploy-local-argocd 

Get password for UI.

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Run

kubectl port-forward service/datapains-argocd-server -n argocd 8080:443

You can now login into the UI with:

ArgoCD UI

• user: admin
• password:

Applications

We have two applications we want to deploy in this workshop.

1. metastore
2. trino

DataPains Trino K8S

We will deploy trino through argocd, the repo we have available on datapains-trino-k8s

You will have to setup a secret key.

Create your secret based on your github SSH key.

 make create-secret REPO_URL=git@github.com:Thelin90/datapains-trino-k8s.git SECRET_NAME=datapains-trino-k8s-creds SSH_KEY_PATH=<path-to-id_rsa>

NOTE!

Remember to build the hive metastore image, instructions here.

Now deploy the metastore:

make apply-argocd-app REPO_NAME=datapains-trino-k8s APP_NAME=metastore

Now deploy trino, once you see it being healthy in ArgoCD UI.

make apply-argocd-app REPO_NAME=datapains-trino-k8s APP_NAME=trino

All set!

Apache Superset

make create-secret REPO_URL=git@github.com:Thelin90/datapains-bi-tools.git SECRET_NAME=datapains-bi-tools-creds SSH_KEY_PATH=<path-to-id_rsa>

make apply-argocd-app REPO_NAME=datapains-bi-tools APP_NAME=superset

Port Forward

kubectl port-forward svc/superset 8088:8088 -n superset

Airbyte

make create-secret REPO_URL=git@github.com:Thelin90/datapains-airbyte.git SECRET_NAME=datapains-airbyte-creds SSH_KEY_PATH=<path-to-id_rsa>

make apply-argocd-app REPO_NAME=datapains-airbyte APP_NAME=airbyte

Note deployment can take a few minutes, check in argocd UI!.

Access UI

UI is now accessible via: wwww.localhost:32767

Spark Operator

First add spark namespace.

kubectl create namespace spark

make apply-argocd-app REPO_NAME=datapains-spark-operator APP_NAME=spark-operator

Argo Workflow

make create-secret REPO_URL=git@github.com:Thelin90/datapains-argo-workflow.git SECRET_NAME=datapains-argo-workflow-creds SSH_KEY_PATH=<path-to-id_rsa>

make apply-argocd-app REPO_NAME=datapains-argo-workflow APP_NAME=argo-workflow

Note deployment can take a few minutes, check in argocd UI!.

Access UI

UI is now accessible via: wwww.localhost:32767