Customization via Bootstrap Scripts for Base K8s Cluster

By Anna V. April 1, 2025. Revised. Version: 0.0.03

1. Introduction

When spinning up a fresh Kubernetes cluster—be it a standard upstream K8s or a K3s instance—organizations often need to immediately apply a set of critical configurations. These can include:

  1. Custom Namespaces and RBAC roles

  2. CNI or Networking policies

  3. Storage Classes and Persistent Volume Claims

  4. Default resource quotas or mandatory Pod Security Standards

To streamline these tasks and ensure every cluster is initialized consistently, you can write bootstrap scripts that run post-cluster creation. The scripts can be triggered automatically by your CI/CD pipeline, or manually by an operator. This document explains how to design these bootstrap scripts, what to include, and how to integrate them with the rest of the k8or Orbit environment.

2. Why Bootstrap Scripts?

  1. Standardization & Repeatability

    • By bundling essential configuration into a single script (or pipeline step), every cluster starts with the same baseline.

    • Minimizes errors and drift from environment to environment.

  2. Time Savings

    • Eliminates the need for operators to manually apply the same YAML or run repeated kubectl commands.

  3. Security & Compliance

    • Ensures each cluster adheres to your organization’s security policies—RBAC rules, Pod Security settings, or custom CNI requirements.

  4. Scalability

    • Automates cluster expansions. In a large environment with many ephemeral or dynamic clusters, automated bootstrap is a must.

3. Core Elements to Include in Bootstrap

Below are common elements your bootstrap scripts might apply:

3.1 Namespaces & RBAC Policies

  • Default Namespaces

    • Dev, Test, Prod, or per-team namespaces.

    • Example:

      apiVersion: v1
kind: Namespace
metadata:
  name: dev
---
apiVersion: v1
kind: Namespace
metadata:
  name: prod

  • RBAC RoleBindings

    • Grant developers read/write access in dev but read-only in prod.

    • Example:

      apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: dev-k8or-role
  namespace: dev
rules:
  - apiGroups: [""]
    resources: ["pods", "services"]
    verbs: ["get", "list", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: dev-k8or-binding
  namespace: dev
subjects:
  - kind: User
    name: "[email protected]"
roleRef:
  kind: Role
  name: dev-k8or-role
  apiGroup: rbac.authorization.k8s.io

3.2 CNI & Networking Policies

  • CNI Customization

    • If you use Calico, you might define custom IP pools or advanced policies.

    • If you use Cilium, you might enable eBPF-based policies.

    • These can be YAML manifests or Helm releases that the script applies.

  • NetworkPolicies

    • E.g., default “deny all” inter-namespace traffic, then allow only what’s needed:

      apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: k8or-deny
  namespace: dev
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

3.3 Storage Setup

  • StorageClass

    • If you have an in-house storage solution or advanced provisioning, define a default StorageClass.

    • Example:

      apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-k80r-storage
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

  • PersistentVolumeClaims

    • The script can also create standard PVC templates for test data or shared logs.

3.4 Resource Quotas & Limits

  • ResourceQuota

    • Limit CPU/memory usage in dev vs. prod.

    • Example:

      apiVersion: v1
kind: ResourceQuota
metadata:
  name: k8or-quota
  namespace: dev
spec:
  hard:
    cpu: "10"
    memory: 16Gi

3.5 Mandatory Pod Security

  • Pod Security Admission or Pod Security Policies (for older K8s versions):

    • Scripts can label namespaces with relevant Pod Security levels (e.g., restricted) or apply custom admission policies.

4. Implementation Approaches

  1. Plain Shell Scripts

    • A simple bash script that runs a series of kubectl apply commands in sequence.

    • Example:

      #!/usr/bin/env bash
kubectl apply -f namespaces.yaml
kubectl apply -f rbac.yaml
kubectl apply -f networkpolicies.yaml
# ...

    • Good for small scale or ephemeral clusters.

    • Potentially integrated into a pipeline or triggered manually.

  2. Helm or Kustomize

    • Keep your bootstrap resources in a single Kustomize overlay or a dedicated Helm chart (e.g., “base-cluster-config”).

    • Example: helm install base-config .

    • More flexible for environment overlays and version control.

  3. Argo CD “Bootstrap”

    • Create a specialized Argo CD application that is set to auto-sync on cluster creation.

    • This approach ensures the cluster’s “base config” is always in sync with your Git repository.

  4. K8Rngr Post-Provisioning Hook

    • If using K8Rngr or a similar orchestrator, you can define a post-provisioning step that triggers your bootstrap script or pipeline.

5. Tying into k8or Orbit Ecosystem

  1. Integration with CI/CD

    • You can store your bootstrap scripts in the same Git repo as your other manifests.

    • The pipeline (e.g., GitLab CI, Jenkins) triggers the script whenever a new cluster is declared or updated.

  2. AccessPoint for Security

    • All kubectl or helm commands from the script must pass through AccessPoint, ensuring RBAC is enforced.

    • The script or pipeline has a service account token with the correct permissions.

  3. ChartStore

    • If any part of your bootstrap involves storing or retrieving specialized chart references, you can keep them in ChartStore (C20).

  4. CommBridge or NetOptimizer

    • If your base config requires custom networking (like route reflection or advanced messaging), you might have the script also apply bridging or route rules in NetOptimizer.

6. Example Implementation (Using Kustomize)

Below is a high-level example that shows how you might structure a “bootstrap” Kustomize project:

bootstrap/
  base/
    namespaces.yaml
    rbac.yaml
    networkpolicies.yaml
    storageclasses.yaml
    # Additional base YAML
  overlays/
    dev/
      kustomization.yaml
      resourcequota-dev.yaml
    prod/
      kustomization.yaml
      resourcequota-prod.yaml

  1. Base directory holds universal resources (e.g., default namespaces, cluster-wide RBAC).

  2. Dev overlay adds or modifies certain quotas, while the Prod overlay might enforce stricter policies or advanced network rules.

Applying this to a new cluster:

kustomize build bootstrap/overlays/dev | kubectl apply -f -

Or, if using Argo CD, define an application referencing this path so it auto-applies to your cluster once it’s recognized by the orbit-plane.

7. Contributors’ Role in Customization

k8or Orbit contributors can provide:

  1. Design & Architecture

    • Evaluate your security, compliance, or performance constraints and propose a base cluster configuration.

    • Outline a recommended set of RBAC rules or network policies.

  2. Implementation

    • Write or refine your bootstrap scripts, Kustomize overlays, or Helm charts.

    • Ensure best practices for handling secrets, storing files in Git, and hooking into the pipeline.

  3. Integration & Testing

    • Incorporate your bootstrap process with cluster creation tools (K8Rngr, ephemeral pipeline scripts, etc.).

    • Validate that the cluster stands up correctly in dev or test environment before going production.

  4. Ongoing Maintenance

    • Provide audits or updates to reflect new compliance policies or additional environment expansions.

    • Keep your bootstrap approach aligned with evolving versions of Kubernetes and orbit-plane components.

PreviousCI/CD Pipelines in k8or OrbitNextMonitoring Customization in k8or Orbit

Last updated