Cluster Bootstrap with Argo CD

By Raj Marni. March 28, 2025. Revisied. Version: 0.0.02

1. Introduction

After the successful creation of a K3s cluster (e.g., dev-k3s) via K8Rngr (C128), the next step is to “bootstrap” the cluster with a set of essential Kubernetes configurations. These configurations ensure that the cluster is ready for application deployments, secure by default, and adheres to organizational policies. Key elements include:

  • Namespaces & Resource Quotas: Segregating workloads and enforcing resource limits.

  • Access Policies: Restricting access to specific namespaces and resources.

  • Calico CNI Customization & Network Policies: Establishing secure and optimized networking configurations.

  • Other Environment-Specific Settings: Such as ingress configurations or logging setups.

All these settings are stored centrally in a DynamoDB table dedicated to configuration management. A dedicated microservice pulls these configurations, converts them into Kubernetes manifest files (YAML or JSON), and then hands them off to Argo CD’s ApplicationSets to deploy them across the new cluster.

2. Configuration Data Source

2.1 DynamoDB as a Central Configuration Repository

  • Configuration Storage: A DynamoDB table is pre-populated with key-value pairs or structured JSON objects that represent the unique configurations for each cluster. This table includes entries for:

    • Namespace definitions (names, labels, annotations).

    • Resource Quotas (CPU, memory limits).

    • Access policies (RBAC rules, role bindings).

    • Calico CNI custom configurations (e.g., IPAM settings, network overlays).

    • Network policies (pod selectors, ingress/egress rules).

    • Additional settings like ingress controller configuration, logging, and monitoring details.

  • Versioning & Overrides: The table is designed to support versioned configurations so that changes can be tracked and new settings can override defaults as needed per environment (dev, test, prod).

3. Bootstrapping Microservice Workflow

3.1 Configuration Retrieval and Conversion

  • Microservice Operation: A dedicated microservice is responsible for:

    • Polling or receiving triggers (e.g., when a new cluster is created) to fetch the required configuration from DynamoDB C20.

    • Parsing the configuration data and converting it into valid Kubernetes manifests (YAML or JSON). This includes templating mechanisms with Kustomize (C28) to fill in dynamic values (such as cluster name, region, etc.).

  • Example Conversion Process:

    • Input: JSON configuration from DynamoDB for a namespace:

      {
  "namespace": "dev-team",
  "labels": { "environment": "dev", "team": "backend" },
  "annotations": { "purpose": "development" },
  "resourceQuota": { "cpu": "4", "memory": "8Gi" }
}

    • Output: A generated YAML manifest:

      apiVersion: v1
kind: Namespace
metadata:
  name: dev-team
  labels:
    environment: dev
    team: backend
  annotations:
    purpose: development
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: dev-quota
  namespace: dev-team
spec:
  hard:
    cpu: "4"
    memory: 8Gi

3.2 Integration with Argo CD ApplicationSets

  • Manifest Delivery: A microservice directly pushes the generated manifest files to a designated Git repository (Argo CD monitoring) or directly hands over the raw JSON/YAML objects via an API call to SyncMaster. (Depending on design requirements.)

  • Argo CD ApplicationSets: Argo CD’s ApplicationSet feature is used to automate the deployment of these manifests:

    • Generator: Reads the configuration repository (or receives API input) and generates multiple Application definitions tailored to the specific cluster.

    • Deployment: Argo CD then applies these Application definitions to the target cluster (e.g., dev-k3s), ensuring that all namespaces, quotas, access policies, Calico configurations, and network policies are created and applied consistently.

4. Example Kubernetes Constructs

4.1 Namespace & Resource Quota

  • Namespace YAML: As shown above.

  • Resource Quota YAML: Included in the namespace definition or as a separate object.

4.2 Access Policies

  • RBAC Example: A Role and RoleBinding manifest:

    apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: dev-team
  name: dev-role
rules:
- apiGroups: [""]
  resources: ["pods", "services"]
  verbs: ["get", "list", "watch", "create", "update", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: dev-role-binding
  namespace: dev-team
subjects:
- kind: User
  name: "[email protected]"
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: dev-role
  apiGroup: rbac.authorization.k8s.io

4.3 Calico CNI Customization

  • Custom Calico Config YAML: An example might define IP pools, network policies, or BGP configurations:

    apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: dev-ippool
spec:
  cidr: 192.168.0.0/16
  ipipMode: Always
  natOutgoing: true
  disabled: false
---
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: allow-dev-traffic
  namespace: dev-team
spec:
  selector: all()
  ingress:
  - action: Allow
    source:
      selector: "environment == 'dev'"

4.4 Network Policies

  • Network Policy Example: Controls traffic between pods:

    apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: restrict-traffic
  namespace: dev-team
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: frontend

5. Data Flow & Process IDs

  • Process IDs (PIDs): Each interaction between the microservice, DynamoDB, and Argo CD is tagged with a PID for traceability (e.g., ms-1234-dynamodb-e10, ms-1234-argoCD-e20).

  • Workflow Summary:

    1. Trigger: New cluster dev-k3s creation triggers the bootstrapping process.

    2. Configuration Pull: The microservice pulls configuration data from DynamoDB.

    3. Manifest Generation: Converts configurations into Kubernetes manifests.

    4. Argo CD Deployment: ApplicationSet generator in Argo CD picks up the manifests and deploys them to dev-k3s.

    5. Confirmation & Logging: Logs are updated in the ChartStore and ViewPort, and status is reported back to the Portal.

6. Error Handling & Rollback Mechanisms

  • Validation Failures:

    • If configuration data is missing or malformed, the microservice logs an error and aborts the manifest generation, triggering an alert.

  • Deployment Errors:

    • Argo CD monitors for unsuccessful ApplicationSet deployments. In such cases, it reverts to the last known good configuration.

  • Audit Logging:

    • Each step (from pulling configuration to applying manifests) is logged, enabling rollback or manual intervention if needed.

7. Benefits & Impact

  1. Consistent Cluster Bootstrapping:

    • Automates the creation of Namespaces, RBAC policies, Calico configurations, and network policies, ensuring each new cluster is pre-configured with standardized best practices.

  2. Dynamic, Environment-Specific Configurations:

    • Unique configurations per cluster are stored in DynamoDB and can be updated without manual intervention.

  3. Seamless Integration with Argo CD:

    • The use of Argo CD ApplicationSets ensures that all configurations are deployed atomically, reducing the chance of configuration drift.

  4. Improved Security and Governance:

    • Predefined access policies, quotas, and network restrictions are applied automatically, ensuring compliance from the outset.

PreviousCluster Creation with K8RngrNextCD with Argo CD

Last updated