Product Customization in k8or Orbit

By Anna V. April 1, 2025. Revised. Version: 0.0.02

API Gateways, Container Runtimes, Databases, Key Management, and Chaos Engineering and Several Other Categories.

1. Introduction

While a base K8s cluster often meets general deployment needs, certain specialized product categories can be crucial for specific organizational requirements. Whether it’s securing API traffic with a gateway, adopting a specific container runtime, or implementing advanced chaos engineering to test resilience, k8or Orbit can integrate these open source products in a consistent and automated manner.

This document outlines the steps, considerations, and best practices for each category and how to weave them into your cluster using the orbit-plane’s existing configuration pipelines (Helm, Kustomize, Argo CD, etc.).

2. API Gateways

Why API Gateways?

  • Enforce advanced routing rules (blue/green, canary releases).

  • Provide centralized authentication, rate limiting, and TLS termination.

  • Observability: capturing request logs, metrics, or tracing.

Popular Open Source Solutions

  1. Kong

    • Extensible with Lua-based plugins, wide community.

    • Good for high traffic, includes built-in load balancing and service discovery.

  2. NGINX Ingress Controller

    • Commonly used as an ingress solution in Kubernetes, can add advanced rewrites, SSL termination.

  3. Traefik

    • Focused on dynamic configuration, with built-in Let’s Encrypt support and easy integration with Docker labels or CRDs.

Integration Steps in k8or Orbit

  1. Deployment with Helm or Kustomize

    • The orbit-plane’s CI/CD triggers chart installation for your gateway of choice.

    • Example: helm install kong/kong --values my-values.yaml.

  2. Configure Gateway CRDs

    • For Kong or Traefik, define CRDs that handle routes, services, authentication plugins.

  3. Secrets & Certificates

    • Store certificates in a secure location (e.g., Vault, k8s secrets). Let the gateway retrieve them from your dedicated K8s secrets or external store.

Where Contributors Help

  • Designing advanced route rules (blue/green or canary).

  • Integrating the gateway with AccessPoint for role-based traffic control.

  • Setting up performance monitoring or custom plugin development.

3. Container Runtimes

Why Container Runtime Customization?

  • Some orgs need specialized runtimes (e.g., containerd, CRI-O) for performance, security features, or resource footprints.

  • Comply with existing on-prem standards or HPC (High Performance Computing) workloads.

Common Runtimes

  1. Docker (Moby)

    • Historically the default, but it’s being replaced in K8s 1.20+ by containerd or CRI-O.

  2. containerd

    • Lightweight, used under Docker’s hood, now widely adopted as a direct CRI (Container Runtime Interface) implementation.

  3. CRI-O

    • A minimal container runtime focusing on open standards, used in Red Hat’s ecosystems.

Integration Steps

  1. Cluster Provisioning

    • At cluster creation time (via K8Rngr or Terraform), specify the container runtime (e.g., containerd).

    • For K3s, it typically uses containerd by default; you can override config if needed.

  2. Security Hardening

    • If you need advanced solutions like SELinux or AppArmor, adjust container runtime configs and node OS settings.

  3. Pipeline Adjustments

    • Ensure your CI pipeline builds images that are runtime-agnostic if possible. If HPC workloads need specialized runtimes, incorporate that into your pipeline scripts.

Where Contributors Help

  • Evaluate performance differences between containerd and CRI-O for your workloads.

  • Provide best practices for HPC or GPU-based containers.

  • Integrate runtime configurations with cluster bootstrap scripts.

4. Databases (In-Cluster or External)

Why Customizing Database Setup?

  • Some workloads require ephemeral test DBs or persistent external DB instances with advanced replication.

  • Data persistence and schema migrations must align with organizational standards (e.g., backups, encryption).

Popular Solutions

  1. PostgreSQL Operators

    • CrunchyData, Zalando, or community operators for Postgres.

  2. MySQL Operators

    • MySQL or MariaDB operators for automating DB creation, backups, or cluster-scale replication.

  3. NoSQL

    • For Cassandra, MongoDB, or DynamoDB-like setups, operators or bridging external services can be integrated.

Integration Steps

  1. Operator Installation

    • Deploy a Helm chart or Kustomize overlay for your chosen DB operator.

    • Example: helm install postgresql-operator . in your environment repo.

  2. CRD Definition

    • Create CRs (Custom Resources) that specify DB size, backups, or user credentials.

    • Possibly store or generate secrets via Vault or a k8s secret to handle DB passwords.

  3. CI/CD Pipeline

    • A pipeline stage might auto-provision ephemeral DBs for test environments, cleaning them up post-tests.

Where Contributors Help

  • Reviewing high availability or multi-AZ designs.

  • Integrating with NetOptimizer if you need multi-region DB traffic routing.

  • Ensuring secure secrets distribution with your existing KMS or Vault.

5. Key Management (Secrets & Certificates)

Why Key Management?

  • Encrypted data, secure TLS certificates, and identity tokens require robust storage and rotation.

  • Meeting compliance demands (PCI-DSS, HIPAA) often requires advanced auditing and centralized management.

Tools & Approaches

  1. HashiCorp Vault

    • Dynamic secret provisioning, automatic rotation of DB passwords, PKI certificate issuance.

  2. Sealed Secrets

    • Encrypts secrets via public key, storing them safely in Git while only being decrypted in the cluster.

  3. KMS Integration

    • Leverage AWS KMS, GCP KMS, or Azure Key Vault for storing encryption keys used by your cluster.

Integration Steps

  1. Deploy a Secrets Operator

    • If using Sealed Secrets or Vault Operator, apply the CRDs.

    • Store your organizational public keys for encryption or provisioning tokens for Vault access.

  2. Update Pipeline

    • Replace plaintext secrets in your Helm charts with references to Vault or sealed secrets.

    • The pipeline retrieves these secrets from a secure location, injecting them at deployment time.

Where Contributors Help

  • Architecting zero-trust or “no plaintext secrets in Git” workflows.

  • Creating secure pipelines that automatically fetch and rotate secrets.

  • Auditing compliance with your existing identity providers.

6. Chaos Engineering

Why Chaos Engineering?

  • Evaluate how your system responds to failures—node crashes, network latency, or container resource exhaustion.

  • Build resilience by systematically injecting failures and measuring service continuity.

Tools

  1. Chaos Mesh

    • Kubernetes-native chaos injection framework for pods, networks, and IO disruptions.

  2. Litmus

    • Another popular framework offering chaos experiments (pod delete, container kill, CPU hog, etc.).

  3. Istio Fault Injection

    • If using Istio, you can define virtual service rules that artificially inject delay or return errors.

Integration Steps

  1. Install the Chaos Tool

    • Deploy Chaos Mesh or Litmus in your cluster with Helm.

    • helm install chaos-mesh ..., for instance.

  2. Define Chaos Experiments

    • Create CRs specifying the chaos scenario, e.g., “kill 1 random dev-namespace pod every 2 hours.”

    • Optionally link them to CI/CD tests or scheduled windows.

  3. Monitor & Observability

    • Use the Orbit plane’s observability stack (ClusterWatch, HelloScope) to measure the effect.

    • Alert if certain SLOs degrade beyond an acceptable threshold.

Where Contributors Help

  • Designing chaos experiments that align with your SLA requirements.

  • Integrating results into CI pipelines (failing a build if resilience is below standard).

  • Setting up “chaos days” or specific intervals for planned resilience testing.

7. Takeway

Product Customization in the k8s environment often spans these categories:

  1. API Gateways – to handle advanced routing, security, and traffic shaping.

  2. Container Runtime – to optimize or secure container execution.

  3. Databases – to manage data in-cluster or seamlessly connect external DB solutions.

  4. Key Management – to handle encryption keys, secrets, and compliance demands.

  5. Chaos Engineering – to actively test and improve cluster resilience.

Each approach requires:

  • Planning (determining which open source solution suits your scenario)

  • Implementation (deploying via Helm, Kustomize, or operator patterns)

  • Integration (tying into the orbit-plane’s CI/CD, logging, and AccessPoint)

  • Maintenance (ongoing version upgrades, security patches, and expansions)

k8or Orbit contributors are poised to help:

  1. Assess your environment and organizational needs,

  2. Design an integration strategy for each product category,

  3. Implement the solutions into your cluster’s bootstrap or continuous deployment process,

  4. Support operational tasks like updates, monitoring, or advanced custom features.

PreviousNetwork Customization in k8or OrbitNextAgreement

Last updated