Network Customization in k8or Orbit

By Anna V. April 1, 2025. Revised. Version: 0.0.02

1. Introduction

Networking in Kubernetes is inherently complex. In k8or Orbit, we strive to unify and automate many aspects of this complexity via default configurations for CNI (Container Network Interface), DNS resolution, and optional service mesh capabilities. However, not all organizations have the same constraints or workflow patterns. Some might need advanced encryption, specialized multi-tenant setups, or unique DNS structures. That’s where network customization plays a vital role.

k8or Orbit is designed to integrate open source solutions like Calico, Cilium, CoreDNS, and Istio. In many cases, these tools can be extended or overridden with custom logic, configuration overlays, or additional security modules to suit the specific needs of an organization. This document serves as a detailed roadmap on how these customizations can be achieved, the typical process flow, and how k8or Orbit contributors can assist.

2. Why Network Customization Matters

  1. Regulatory & Compliance

    • Certain industries (finance, healthcare, government) demand strict traffic controls (e.g., HIPAA, PCI-DSS). A custom network layout or advanced encryption may be needed.

  2. Performance Tuning

    • Organizations with high throughput demands might tweak their CNI plug-ins for optimized route reflection, low-latency bridging, or advanced eBPF usage.

  3. Multi-Cluster, Multi-Cloud

    • Complex topologies across multiple clusters or different cloud providers may require advanced load balancing, custom DNS resolution, or unified cross-cluster networking (mesh).

  4. Unique IPAM Requirements

    • Some companies demand certain IP ranges or hierarchical IP allocation to align with on-prem or cloud VLAN constraints.

3. Components Involved

3.1 CNI Plug-ins (e.g., Calico, Cilium)

  • Calico: Known for its robust policy model, IP-in-IP or VXLAN overlays, and easy integration with network policies.

  • Cilium: Provides advanced eBPF-based networking, enabling deeper observability and potential performance gains for layer 7 policies.

Customization Examples:

  • Custom IP Pools: Allocate or restrict certain IP ranges for dev vs. prod.

  • Policy Overlays: Insert additional custom network policies or advanced features (like eBPF) for traffic shaping.

3.2 CoreDNS

  • Primary DNS for Pods

    • Typically used to resolve service names, pod hostnames, and external resources.

  • Customization:

    • Add custom domain forwarding rules.

    • Set up advanced DNS filters or caching.

    • Integrate with internal DNS servers for cross-data-center name resolution.

3.3 Istio (or other Service Mesh)

  • Advanced Traffic Management

    • Capable of zero-trust security, mutual TLS, advanced routing, traffic splits, canary deployments, etc.

  • Customization:

    • Use custom Envoy filters to handle specialized protocols.

    • Integrate with in-house PKI or identity providers for secure mTLS.

    • Extend telemetry pipelines for custom metrics or logs.

3.4 NetOptimizer in k8or Orbit

  • Meta-Component

    • Aggregates multiple connectivity components (e.g., wired/wireless adapters, NAT gateways, advanced overlay logic).

  • Customization:

    • Introduce specialized route reflection, bridging, or cross-VPN logic.

    • Automate multi-cloud or multi-region routing policies with your own BGP or GRE tunnels.

4. Approaches to Customizing Network Components

  1. Helm / Kustomize Overlays

    • Each open source tool (Calico, Istio, etc.) often has a Helm chart or Kustomize package.

    • k8or Orbit uses these by default, but you can add environment overlays or custom values to inject advanced configurations.

    • E.g.:

      # Example Kustomize overlay for custom Calico IP pool
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: k8or-ippool
spec:
  cidr: 10.10.0.0/16
  natOutgoing: true
  ipipMode: Always

  2. Custom CRDs & Operators

    • If your organization has unique networking requirements, you might create a custom CRD (e.g., NetworkSegment or GlobalDNSRule) and an operator that enforces or syncs these to the underlying CNI or DNS config.

    • k8or Orbit can load these CRDs during cluster bootstrapping or post-creation, ensuring consistent network rules.

  3. Integration with K8Rngr / Argo CD

    • K8Rngr can pass specialized cluster network parameters during cluster creation (like specifying a particular CNI or custom IP ranges).

    • Argo CD can handle the continuous deployment of network policy manifests or Istio config changes across dev, test, and prod clusters.

  4. Plugin Mechanisms & Sidecars

    • You can embed a sidecar or a plugin that intercepts or modifies traffic for advanced use cases (like hooking into eBPF or layering encryption).

    • Similar to how a “wrapper” approach is used for sidecar proxies, you can implement a sidecar that loads custom IPTables or eBPF rules.

5. Process Flow for Organizational Customization

Below is a recommended step-by-step approach:

5.1 Requirements Gathering

  1. Identify what is missing from the default networking setup.

    • Do you need custom IP pools? Custom DNS forwarders? eBPF-based security policies?

    • Are there compliance constraints or advanced performance needs?

  2. Scope the changes.

    • Are they cluster-wide or environment-specific?

    • Do you plan to replicate them for dev, test, and prod?

5.2 Prototyping & Testing

  1. Local Test Environment

    • Spin up a local K3s or kind cluster.

    • Experiment with Calico or Istio overlays in a controlled setting.

  2. Validation

    • Validate that your custom config works (e.g., pings are allowed, DNS resolves, or traffic splits are correct).

5.3 Integration with k8or Orbit

  1. Helm / Kustomize Definitions

    • Place your final configuration files (like advanced IP pools or custom DNS stubs) in your Git repository.

    • Leverage k8or Orbit’s Argo CD pipelines to apply these automatically upon cluster creation or updates.

  2. K8Rngr Configuration

    • If changes are needed at cluster provisioning time (like specifying a custom CNI or IP range), you can configure K8Rngr’s cluster creation parameters to incorporate these values.

  3. Observability & Logging

    • Integrate your custom changes with the k8or Orbit logging and monitoring stack (ClusterWatch, HelloScope), ensuring you track any performance or error metrics for your custom network flows.

5.4 Ongoing Maintenance

  1. Document changes thoroughly for your ops teams.

  2. Monitor network performance and security posture to refine or update custom logic as your environment grows.

  3. Periodic Audits with k8or Orbit’s AI agents or manual checks to ensure no drift in custom config across dev, test, and prod.

6. How k8or Orbit Contributors Can Help

  1. Assessment & Design

    • Contributors can review your existing network architecture and propose a design that extends or modifies open source networking solutions.

    • They identify optimal ways to integrate with k8or Orbit’s environment overlays, CI/CD pipelines, and cluster lifecycle management.

  2. Implementation & Testing

    • They write or refine your Helm/Kustomize overlays, CRDs, or custom code to tailor the network.

    • They set up test clusters in a staging environment to ensure your network customizations function as intended.

  3. Deployment & Ongoing Support

    • Once the design passes validations, contributors can help implement the new networking configs across your clusters, hooking them into K8Rngr, Argo CD, or NetOptimizer.

    • They provide training and documentation to your ops teams for daily management, plus ongoing assistance if updates or expansions are required.

  4. Integration with AI Agents

    • For advanced setups, contributors can integrate or adapt the built-in AI agents (e.g., Pydantic-based sidecars) to “understand” and manage your specialized network rules, offering code-level help or troubleshooting guidance.

7. Example Use Cases

  1. Encrypted Pod-to-Pod Communication

    • A finance company adds an IPsec or wireguard layer within Calico for in-transit encryption, integrated via custom chart overlays and orchestrated by Argo CD.

  2. Multi-Tenant DNS

    • A hosting provider modifies CoreDNS to forward specific domain queries to private DNS servers per tenant. They manage these rules via a custom CRD that the orbit plane applies on cluster creation.

  3. Istio mTLS

    • A healthcare company enforces mutual TLS among microservices for HIPAA compliance, with specialized Istio policies automatically injected at cluster bootstrap.

  4. Cross-Cloud VPC Peering

    • A gaming studio needs multi-cloud routing. They adapt NetOptimizer to handle route reflection and NAT gateways, allowing pods in different clouds to communicate seamlessly.

PreviousMonitoring Customization in k8or OrbitNextProduct Customization in k8or Orbit

Last updated