Enforcing Policies with OPA Gatekeeper
Context
The configuration of Kubernetes suffers from several vulnerabilities. For instance, any developer can escalade his privileges using this tutorial
With k8saas, the common risks are reduced by builtin security mechanisms, like:
- all accesses are managed using Central Authentication (Azure Active Directory with Multiple Factor Authentication Enforced).
- Role Base Access Control (RBAC) is enabled on Kubernetes. Meaning, no-one has full admin access to the cluster (except level 2 / core team).
- By design, the cloud resources are dedicated to a project (logical isolation and limited transversal permissions). Rebound to another cluster is very complex.
To improve the security of our clusters, k8saas has enforced policies.The management of these policies has moved from Azure Policy for AKS to OPA gatekeeper
Use case
- Perform a pentest to test the robustness of k8saas
- Protection against credentials steeling
- Protection against malware in kubernetes
What to do ?
Nothing ! The service is enabled by default.
Concept
Before, the k8saas policy was based on Azure Policy for AKS. The rational was to use "ready-to-use" feature properly integrated with Azure.
Azure portal provide IHM to manage policies.
Azure Policy for AKS uses gatekeeper of the Open Policy Agent Project.
Limitations:
- Azure Policy for AKS supports officially only azure builtin rules
- Azure Policy for AKS is complex to debug. During the engineering phase, the iterations takes too many times
K8saas has decided to use the same product (behind azure policy for AKS), ie Gatekeeper; but deployed, configured and monitored directly by k8saas team.
List of policies enabled by default for k8saas:
- no-privileged-container
- block-node-selector
- ingress-host-max-lenght
- nginx-ingress-classname
List of available policies:
| k8saas policy name | Description | Enable by default |
|---|---|---|
| No-privileged-container | Controls the ability of any container to enable privileged mode. Corresponds to the privileged field in a PodSecurityPolicy. For more information, seehttps://kubernetes.io/docs/concepts/policy/pod-security-policy/#privileged | true |
| all-ns-must-have-gatekeeper-label | Put a gatekeeper label (tutorial) | false |
| block-node-selector | refuse pod with node-selector | true |
| ingress-host-max-length | Limit the host length when submitting ingress objects | true |
| containers-shall-only-use-allowed-images | limit the usage of specific pre-allowed artifact repositories | false |
| nginx-ingress-classname | Block the legacy ingress class name annotation | true |
HOWTO
Disable the policy for a specific cluster ?
Please send a request on the TrustNest K8SaaS Service catalog with:
- the cluster used by the project
- the reason
- the contact of your security partner & Product Owner
Ask for a new policy ?
Please send a request on the TrustNest K8SaaS Service catalog with:
- the cluster used by the project
- the list of policies to enabled
- the contact of your security partner & Product Owner
Monitor my policies
There are 2 useful dashboard that you can use to monitor your policies:
- K8saas / K8SAAS / OPA Gatekeeper
This dashboard shows you the type of policies deployed on your cluster (deny, warn etc...), and if they have blocked some deployment on your cluster.
- Flux Cluster Stats
This dashboard shows you the kustomizations deployed on your cluster, including the policies. With the naming convention and the description of the available policies (table just before) you can understand what are the deployed policies.
Especially, on the left table:
TROUBLESHOOTING
How to know when I'm blocked by a policy ?
When submitting a pod (manually or from a deployment/statefulset), you should see a log like:
Error from server ([denied by azurepolicy-container-no-privilege-6134f47be9a69edbacdd] Privileged container is not allowed: noderootpod, securityContext: {"privileged": true}
[denied by azurepolicy-container-no-privilege-7f83897b6cbbf9956e71] Privileged container is not allowed: noderootpod, securityContext: {"privileged": true}): error when creating "noderoot.yml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by azurepolicy-container-no-privilege-6134f47be9a69edbacdd] Privileged container is not allowed: noderootpod, securityContext: {"privileged": true}
[denied by azurepolicy-container-no-privilege-7f83897b6cbbf9956e71] Privileged container is not allowed: noderootpod, securityContext: {"privileged": true}
Those logs could be in the events:
$ kubectl get events -n <your_namespace> -w