Skip to main content

Deploy with gitops

info

Feature available with Dalim 4.X+ on early adopter

Manage your deployment with flux

GitOps on k8saas subscription is based on flux. For a short introduction GitOps it's like doing DevOps with git as source of truth. Flux work as a pull-based system within your k8saas cluster. This core functionality allows your system to always be in sync with the state described in a Git repository.

Putting in place gitops approach requires to solve some challenges:

  • Security: GitOps revolves around using Git as the central source of truth, which means sensitive data like secrets might end up being stored in Git. It's crucial to encrypt those secrets and manage decryption keys securely.
  • Storing Secrets: Secrets should never be stored in plaintext in the git repository. Always use a secure method to store credentials. Kubernetes provides Secrets as a resource, or third-party solutions exist such as Helm Secrets, AWS Secrets Manager, or HashiCorp Vault.
  • Segregation of Permissions: It's crucial to manage roles and responsibilities precisely. Implementing a Role-Based Access Control (RBAC) model can help differentiate permissions based on job requirements. Also, use the least privilege principle where users get minimum levels of access necessary to perform their jobs.
  • Monitoring & Troubleshooting: Using a monitoring tool will help you keep track of system health and detect anomalies or security incidents. Set up logging for auditability of actions and system state. If anything goes wrong, logs can be a vital resource for troubleshooting and finding remediation steps.
  • Repository architecture: A mono-repo or multi-repo style can be used, each having their own pros and cons.

Getting Starterd

Setup tools

  1. Install kustomize
  • on linux or macos Install kustomize
brew install kustomize
  • on windows
choco install kustomize
  1. Install flux cli
  • on linux or macos Install flux cli
brew install fluxcd/tap/flux
  • on windows
choco install flux
  1. Install sops

follow official documentation: https://github.com/getsops/sops/releases

Demo repositories

repositories used for following demonstration are located here: https://gitlab.thalesdigital.io/platform-team-canada/k8saas-innersource/gitops-samples

First step with gitops

This sample demonstrate a simple use of flux with one repository and 2 environments dev and qa

  1. Create fork project https://gitlab.thalesdigital.io/platform-team-canada/k8saas-innersource/gitops-samples/basic-sample
  • create access token with required permissions

    • scope: read_repository
    • role: maintainer
  • create access token

    • export value into GIT_TOKEN
export GIT_TOKEN=XXXXX
[You can also create access token at group level]
  1. Inspect sample repository

  2. repository should look like:

   ├── apps
│   └── simple-hello-world
│   ├── configmap.yaml
│   ├── ingress.yaml
│   ├── kustomization.yaml
│   └── manifest.yaml
└── envs
├── dev
│   ├── kustomization.yaml
│   └── patches
│   └── patch-ingress.yaml
└── qa
├── kustomization.yaml
└── patches
├── patch-config.yaml
├── patch-deployment.yaml
└── patch-ingress.yaml

Description of repository

  • apps: This folder contains the infrastructure of application simple-hello-world
    • simple-hello-world: application folder.
      • configmap.yaml: This file defines the ConfigMaps for application.
      • ingress.yaml: This file defines the Ingress for application.
      • manifest.yaml: This file contains the manifests of application
      • kustomization.yaml: This file identify all object that will be rendered
  • envs: This folder contains the specific environments of your project.
    • dev: default configuration
    • qa:
      • kustomization.yaml: This file is used by Kustomize to customize the configurations for the specific environment.
      • patches: This folder contains patch files that are used to modify resources for a specific environment.
        • patch-ingress.yaml, patch-config.yaml, patch-deployment.yaml: These files contain environment-specific modifications for the corresponding resources.
  1. Setup flux git repository
  • create git secret
kubectl create secret generic -n customer-namespaces git-token --from-literal=username=gittoken --from-literal=password=$GIT_TOKEN
  • create git repository
flux create source git -n customer-namespaces hello-world-sample --url <fork_git_url> --branch=main --secret-ref=git-token
  • --url: flag specifies the URL of the Git repository. You should replace <fork_git_url> with the actual repository path that you want to refer to.

  • --branch flag tells Flux to track the main branch of the Git repository.

  • --secret-ref flag is used to refer to the Kubernetes Secret git-token that contains credential details used to authenticate against the Git repository.

  • Verify information about source repository

flux get source git -n customer-namespaces hello-world-sample
  1. Init dev namespace

Before deploying application with flux some prerquisites are required to allow flux controller to deploy application.

  • Create dev namespace and assign devops cluster role to service account k8saas-generic-sa-cicd
kubectl hns create dev -n customer-namespaces
kubectl create rolebinding gitops-rolebinding --clusterrole devops-namespace-role --serviceaccount=customer-namespaces:k8saas-generic-sa-cicd -n dev
  • Edit configuration
    • Into forked repository edit the file patches/patch-ingress.yaml in dev environment to replace CLUSTER-NAME by the name of your k8saas instance.
    • Then check modifcation by running kustomize command:
kustomize build . --load-restrictor LoadRestrictionsNone   | grep "host:
  • finaly commit and push your changes

  • setup dev environment

flux create ks -n customer-namespaces  dev-system --source=GitRepository/hello-world-sample--path envs/dev --service-account k8saas-generic-sa-cicd --prune true --target-namespace dev
  • --source flag is used to specify the source for the Kustomization. In this instance, it's using a GitRepository source with the name 'hello-world-sample'.

  • --path flag specifies the path in the aforementioned Git repository where the Kustomization manifests are stored. Here, it is pointing to the 'envs/dev' directory.

  • --service-account flag is used to specify the name of the Kubernetes Service account to be used for this Kustomization.

  • --prune flag is used to remove resources from the cluster that are not present in the Git repository. This flag ensures that the cluster state always mirrors the state described in Git.

  • --target-namespace flag specifies the namespace to which the resources will be applied in the cluster.

  • Verify kustomization state

flux get ks  -n customer-namespaces  dev-system
  • Verify object deployed by kustomization

With flux cli:

flux tree ks -n customer-namespaces dev-system

With kubectl:

kubectl get ks  -n customer-namespaces dev-system -o jsonpath="{.status.inventory}" | jq
  • browse the ingress
  1. Setup qa namespace
  • Create qa namespace ans assign devops cluster role to k8saas-generic-sa-cicd
kubectl hns create qa -n customer-namespaces
kubectl create rolebinding gitops-rolebinding --clusterrole devops-namespace-role --serviceaccount=customer-namespaces:k8saas-generic-sa-cicd -n qa
  • Edit configuration
    • Into forked repository edit the file patches/patch-ingress.yaml in qa environment to replace CLUSTER-NAME by the name of your k8saas instance.
    • Then check modifcation by runing kustomize command:
kustomize build . --load-restrictor LoadRestrictionsNone   | grep "host:
  • finaly commit your changes
  • setup qa nevironment
flux create ks -n customer-namespaces  qa-system --source=GitRepository/hello-world-sample--path envs/qa   --service-account k8saas-generic-sa-cicd --prune true --target-namespace qa
  • Verify kustomization state
flux get ks  -n customer-namespaces  qa-system
  • Verify object deployed by kustomization

With flux cli:

flux tree ks -n customer-namespaces qa-system

With kubectl:

kubectl get ks  -n customer-namespaces qa-system -o jsonpath="{.status.inventory}" | jq
  1. browse the ingress
  1. Conclusion:

In this basic sample we've shown:

  • how to prepare k8saas environment to use gitops
  • how to interact with flux with cli
  • how to use overlay to override some values with kustomize

As you may notice, we did not use any git flow in this section. Indeed monorepo does not make it easy to use the git flow approach, as environment are managed by folder and can introduce drifts between branches. In following section we will discuss of different approach to structure git repo to have better separation of concern between applications infrastructure and configuration. Secret management with gitopsis antother key point that has not been discussed

Git topologies

Coming Soon

Work with flux

Flux objects

the main flow resources used in a project are:

1.Source

Source represent a repository where your applications infrastructure or configuration redise. Often it's a git repository. Flux support other type of repositories:

  • bucket: allow to declate bucket as the source. In case of k8saas, bucket is blob stotage account
     ---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: Bucket
metadata:
name: azure-service-principal-secret
namespace: default
spec:
interval: 5m0s
provider: azure
bucketName: <bucket-name>
endpoint: https://<account-name>.blob.core.windows.net
secretRef:
name: azure-sp-auth
---
apiVersion: v1
kind: Secret
metadata:
name: azure-sp-auth
namespace: default
type: Opaque
data:
tenantId: <BASE64>
clientId: <BASE64>
clientSecret: <BASE64>
  • gitrepository: use git as source
     apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: podinfo
namespace: default
spec:
interval: 5m0s
url: https://github.com/stefanprodan/podinfo
ref:
branch: master
 - helmrepository: use helm2 repository or oci
       apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: bitnami
namespace: flux-system
spec:
interval: 10m
url: https://charts.bitnami.com/bitnami
 - helmchart: helmchart is not used directly.  helmchart object is used in helmrelease object below chart property.
       apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: frontend
namespace: default
spec:
interval: 5m
chart: # <<< represent helmchart object
spec:
chart: podinfo
version: ">=4.0.0 <5.0.0"
sourceRef:
kind: HelmRepository
name: podinfo
namespace: default
interval: 1m
  - ocirepositories: use oci repository like azure container registry. For now only kubelet managed idenity is supported
         apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
name: podinfo
namespace: default
spec:
interval: 5m0s
url: oci://ghcr.io/stefanprodan/manifests/podinfo
ref:
tag: latest
  1. Kustomize controller

This controller is in charge of:

  • Reconciles the cluster state from multiple sources (provided by source-controller)
  • Generates manifests with Kustomize (from plain Kubernetes YAMLs or Kustomize overlays)
  • Validates manifests against Kubernetes API
  • Prunes objects removed from source (garbage collection)
  • Impersonates service accounts (multi-tenancy RBAC)
  • Decrypts Kubernetes secrets with Mozilla SOPS and KMS
  • Health assessment of the deployed workloads
  • Runs pipelines in a specific order (depends-on relationship)
  • Reports cluster state changes (alerting provided by notification-controller)

The k8saas installation uses service account impersonation to guarantee isolation between the k8saas deployment and the client. This is why you need to create a service account and associate it with a rolebinding on the namespace in which you want to deploy your application. If no service account is specified, the controller takes the default one. In this setup crossnamepace operation context and all flux object reside in the namespace.

Another point take in consideration kustomize contoller only reconcile objects managed by itself ( found in inventory section )

  1. Helm controller

Work with secrets

Managing secret with gitops in pull mode is one point of attention. GitOps, as a methodology, focuses on using Git as the single source of truth for your infrastructure and application definitions. bellow we describe two approach

  1. With SOPS and shared key SOPS: Secrets OPerationS is an encryption tool that supports multiple format file such as yaml, json, ini format with Azure Key Vault. SOPS use AES256 meaning one encryption key is used to encrypt and decrypt. Integration with azure keyvault allow to operator to encrypt data on their side, push encoded secret into git and at reconcillation time flux decrypt secret and apply manifest if needed.

    In k8saas setup kustomize controller use workload id to access to the key in keyvault.

  • First you need an azure key vault to store the key.
      keyvault="k8saas-keyvault-name"
resource_group="k8saas-cluster-name"
sops_key="sops-key"
az keyvault create -g "$resource_group" -n "$keyvault"
az keyvault key create --name "$sops_key" \
--vault-name "${keyvault}" \
--protection software \
--ops encrypt decrypt
  • Asssign permission decrypt to workload id
     cluster_resource_group=""
IDENTITY_NAME="${cluster_resource_group}-flux-wid"
export IDENTITY_ID="$(az identity show -g ${cluster_resource_group} -n ${IDENTITY_NAME} -otsv --query principalId)"
# depending on permission model, here vault was build with 'Vault access policy'
az keyvault set-policy --name ${keyvault} --object-id ${IDENTITY_ID} --key-permissions decrypt
sops_key_id="$(az keyvault key show --name "$sops_key" --vault-name "$keyvault" --query key.kid -otsv)"
  • Create sops definition file .sops.yaml on root directory of project like
  cat <<EOF >>.sops.yaml
creation_rules:
- path_regex: "envs/.*"
azure_keyvault: "$sops_key_id"
encrypted_regex: ^(data|stringData)$
EOF

In this version we used a kubernetes secret to store the welcome message

  • Encrypt exiting file envs/qa/patches/patch-secret.yaml
sops --encrypt --in-place envs/qa/patches/patch-secret.yaml
  • verify that file is correcly encoded
cat envs/qa/patches/patch-secret.yaml
   apiVersion: v1
kind: Secret
metadata:
name: aks-helloworld-one
data:
title: ENC[AES256_GCM,data:tpRm7Wvqxc5YhzvoziCiyRVh40tbOz14oBSKbVcxdHWlNwpKDNY=,iv:1DZF/7Iyg5hrquN22nbJwQz8V3P+7qvR76kFYTl8aZ tag:ObXaChrurfbYyURthamNgg==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv:
- vault_url: https://XXXXXXXX.vault.azure.net
name: sops-key
version: 3bf1e70f408e4736ac7fadfe67ad2bdc
created_at: "2023-11-13T16:35:34Z"
enc: sdY6KkZTe900dniUIZaNgEE0Q6L-iP9O2gaakHGdQqUbkBeReakdzFKkJ2Z_lma1JcPRaHe0sImiCGT4A15yNvR2z-uewoCe0eNJew7hHQoFWTePvvnzQld7HcZzGO3PU07K OS8KMXiWgej-kQlSvXT_m-gjV azHc1RCFUTMTdJtPVA5dCLsxbGEvYt5d_I9vTCyKyfLzH40bDO-vAaJ89A0luiCFu9PWFEzyx2cbkO0tZOwb6V-UKW6fAYzXbEpXhmAYN6MUSjDJO4Q98vf5FLn4554gWKUs wLcg2xu4HO9GzfnGqWFROSKqa 0bPQcFx8RqhOe0ixmkX7rQ
hc_vault: []
age: []
lastmodified: "2023-11-13T16:35:37Z"
mac: ENC[AES256_GCM,data:X7IwFaXc2wjdiqz1IXMLpkrVd84qGDVffFdead5g8bgJ5C9vvf/lrAQE8w2dYV83es0WKkA1Rmu3agGadLp4LaUmMdnyK oe3vVjHS2iIYGO33uOioOh4UM7qoolJM3lqO4L+zjjeK +pH+ImFVMWsr9u9s8z7pRiH99hcJok4=,iv:NiCQac+FzMy9PtS2Z3KGrU8a2MjX+IbOyIP1Twn0Jpo=,tag:s CeY3xtqfxRsJAw7YAw==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.7.3
  • Commit and push

  • Setup new namespace 'qa-secret'

   ns="qa-secret"
kubectl hns create "$ns" -n customer-namespaces
kubectl create rolebinding gitops-rolebinding --clusterrole devops-namespace-role --serviceaccount=customer-namespaces:k8saas-generic-sa-cicd -n "$ns"
  • setup flux environment

  • Create new git access token

   kubectl create secret generic -n customer-namespaces git-token-secret --from-literal=username=gittoken --from-literal=password=$GIT_TOKEN
  • Create gitrepository
  flux create source git -n customer-namespaces hello-world-sample-secret  --url <fork_git_url> --branch=main --secret-ref=git-token-secret
  • Create environment qa-secret
   flux create ks -n customer-namespaces  "$ns-system" --source=GitRepository/hello-world-sample-secret --path envs/qa   --service-account k8saas-generic-sa-cicd --prune true --target-namespace "$ns" --decryption-provider sops
  • Create port-forward then browse service
  kubectl port-forward -n qa-secret svc/aks-helloworld-one 8081:80

Open url localhost:8081 with a browser

  • Rotate secret in git
  sops envs/qa/patches/patch-secret.yaml
  1. With Azure KeyVault Provider

Azure Key Vault Provider (AKVP) for Kubernetes secret store CSI Driver allows you to get secret contents stored in an Azure Key Vault instance and use the Secrets Store CSI driver interface to mount them into Kubernetes pods. Mounts secrets/keys/certs to pod using a CSI Inline volume.

Major difference with previous method is secrets are no longer managed by git.

Like in previous section we'll need a keyvault to store secret.

  • Add secret into keyvault
   keyvault="vltgitops123"
resource_group="<cluster-name>"
secret_value_base64=$(echo -n "AKVP secret message"| base64)
az keyvault secret set --name secretMessage --vault-name "$keyvault" --value "$secret_value"

Secret has to be encoded in base64 because it will be sync with kubernetes secret at the end

  • Add permission to azure identity to read our secret. An identity <cluster-name>-customer-workload-id is already deployed and usable from customer-namespaces
   IDENTITY_NAME="${cluster_resource_group}-customer-workload-id"
export IDENTITY_ID="$(az identity show -g ${cluster_resource_group} -n ${IDENTITY_NAME} -otsv --query principalId)"
az keyvault set-policy --name ${keyvault} --object-id ${IDENTITY_ID} --secret-permissions get list
  • Now we have defined how secret will be used by the application. Indeed 2 choice are possible:

    1. mount secret directly into application using CSI Inline volume. To used this methode application have to read secret from local file. Main inconvenient of this method when secret is updated into keyvault, application is not aware of changes and has to watch secret for changes
    2. Second option is to sync keyvault secret with kubernetes secret and use kustomize post build feature to reload application. This scenario that will be cover in following
  • Create SecretProviderClass

  IDENTITY_NAME="${cluster_resource_group}-customer-workload-id"
export IDENTITY_ClientID="$(az identity show -g ${cluster_resource_group} -n ${IDENTITY_NAME} -otsv --query clientId)"
export tenantId="$(az account show --query tenantId -otsv)"
cat<<EOF >>secretprovideclass.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: azure-sync
spec:
parameters:
clientID: $IDENTITY_ClientID
keyvaultName: $keyvault
tenantId: $tenantId
usePodIdentity: "false"
useVMManagedIdentity: "false"
objects: |
array:
- |
objectName: secretMessage
objectType: secret
objectVersion: ""
provider: azure
secretObjects:
- data:
- key: title # map secretMessage keyvault secret to title key in aks-helloworld-one secret
objectName: secretMessage
secretName: aks-helloworld-one
type: Opaque
EOF
kubectl apply -n customer-namespaces -f secretprovideclass.yaml
  • Create a deployemt to mount secret, this step is required to sync azure keytvault secret with kubernetes
  cat<<EOF >> secret-sync-deployment.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
name: secret-sync-deployment
spec:
replicas: 1
selector:
matchLabels:
app: secret-sync-deployment
template:
metadata:
labels:
app: secret-sync-deployment
name: secret-sync-deployment
spec:
serviceAccountName: k8saas-customer-sa-workload-id
containers:
- name: busybox
image: docker.io/busybox:1.35.0
resources:
requests:
cpu: 20m
memory: 32Mi
command:
- "/bin/sleep"
- "10000"
volumeMounts:
- name: secrets-store01-inline
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store01-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: azure-sync
EOF
kubectl apply -n customer-namespaces -f secret-sync-deployment.yaml
  • Verify that secret is created under namespace customer-namespaces
kubectl get secret -n customer-namespaces aks-helloworld-one  -ojsonpath="{.data.title}" | base64 -d
  • Under repo gitops-samples-secrets switch to the branch akvp
git checkout -b akvp

review changes

  git diff main..akvp
  • Create new environment named qa-akvp that use branch akvp
  • prepare env
  ns="qa-akvp"
kubectl hns create "$ns" -n customer-namespaces
kubectl create rolebinding gitops-rolebinding --clusterrole devops-namespace-role --serviceaccount=customer-namespaces:k8saas-generic-sa-cicd -n "$ns"
flux create source git -n customer-namespaces hello-world-sample-secret-akvp --url <fork_git_url> --branch=akvp --secret-ref=git-token-secret
  • Create environment
  cat<<EOF >> ks-postbuild-manifest.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: qa-akvp-system
namespace: customer-namespaces
spec:
interval: 1m0s
path: ./envs/qa
prune: true
serviceAccountName: k8saas-generic-sa-cicd
sourceRef:
kind: GitRepository
name: hello-world-sample-secret-akvp
targetNamespace: qa-akvp
postBuild:
substituteFrom:
- kind: Secret
name: aks-helloworld-one

EOF
kubectl apply -n customer-namespaces -f ks-postbuild-manifest.yaml
  • inspect secret into qa-akvp namespace
  kubectl get secret -n qa-akvp aks-helloworld-one -ojsonpath="{.data.title}" |base64 -d
  • Create port-forward then browse service
  kubectl port-forward -n qa-akvp svc/aks-helloworld-one 8081:80

Open url localhost:8081 with a browser

  • Rotate secret in keyvault and observe event in customer-namespaces and qa-akvp

Known limitations

  • Controllers ImageRepository, ImagePolicy, ImageUpdateAutomation are not available
  • not UI to Follow object like argoUI, Weave UI is still undeer experiment on our side

troubleshooting

Error messageSolution
helmreleases.helm.toolkit.fluxcd.io "keda" is forbidden: User "system:serviceaccount:flux-system:default" cannot patch resource "helmreleases" in API group "helm.toolkit.fluxcd.io" in the namespace "flux-system"Add service account name field in the kustomization's spec & the inherited Flux resources
econciliation failed: failed to get last release revision: query: failed to query with labels: secrets is forbidden: User "system:serviceaccount:flux-system:default" cannot list resource "secrets" in API group "" in the namespace "flux-system"Add the service account name spec in your helm release definition file and all the Flux resources
can't access 'XXX/XXX/XXX', cross-namespace references have been blockedAll Flux resources should be defined in the same namespace, but you can specify a deployment destination namespace by adding the spec targetNamespace in your Flux resource definition file

more cheatsheet