[SOON] Split node pool
Context
Until now, K8SaaS operated with a single default node pool to host AKS, K8SaaS, and customer workloads. As customer workloads have matured and the demand for performance and stability has increased, splitting the node pool has become essential.
Objective
The key objectives of this split are to:
- Isolate workloads between K8SaaS and customers.
- Allow customer workloads to scale independently and use their own SKUs.
- Facilitate cost management by optimizing customer workloads.
Implementation
K8SaaS workloads now utilize a preferred affinity to the system node pool. Some components are critical and can affect customer workloads (such as NGINX, HNS, and Gatekeeper); therefore, they may still move to the customer node pool in cases of:
- Migration
- Node saturation
Additionally, DaemonSets will continue to be deployed on the customer node pool to collect logs and metrics.
By design, this feature enables Azure CNI Overlay.
To facilitate pod placement, a new label, k8saas-nickname, has been added to the customer node pool with the value userpool. This label value remains constant during node pool upgrades.
Taints are not added by default, but can be added by support