AWS Controllers for k8or

By Anna V. April 23, 2025. Version: 0.0.01

Layer 1: The Big Picture

What is this?

In the k8or Orbit platform, infrastructure and application deployment are unified under GitOps principles. AWS Controllers for k8or (ACK) extend this paradigm by enabling Kubernetes-native provisioning of AWS services directly from within Orbit-managed clusters.

These controllers make AWS services — like S3, RDS, DynamoDB, and SNS — manageable using Kubernetes manifests, Git repositories, and Orbit’s continuous delivery systems. This turns cloud infrastructure into versioned, declarative, auditable resources governed by the same pipelines that manage Orbit microservices.

Why it matters

• Developers and DevOps teams no longer need to use external tools (Terraform, CloudFormation) to provision cloud infrastructure.

• Resources like S3 buckets or RDS databases can be declared as YAML in the same Git repo as the application that uses them.

• Orbit environments (dev/test/prod) can be dynamically provisioned and consistently managed across all clusters with zero manual AWS console interaction.

---

Layer 2: k8or-Specific Integration Overview

How it works in k8or Orbit

ACK introduces CRDs (Custom Resource Definitions) for AWS services and pairs each CRD with a controller that syncs the desired state to AWS via API calls.

In the Orbit ecosystem, each controller:

• Runs in a K3s cluster under Orbit management.

• Authenticates through IRSA (IAM Roles for Service Accounts) to ensure scoped access per environment.

• Receives configuration via Argo CD (C108), syncing with manifests generated or stored in GitLab (C100) or ChartStore (C20).

• Operates under the security perimeter of AccessPoint (C52) to control and log API interactions.

Orbit-Aware Features

Feature	Orbit Integration
Cluster Bootstrapping	ACK manifests are deployed during post-creation setup.
Continuous Delivery	Managed via Argo CD ApplicationSets.
Environment Isolation	IAM roles and CRDs are namespace-scoped by Orbit env.
Audit & Security	All ACK activity is logged via AccessPoint and ViewPort.

Common Use Cases in Orbit

• Provisioning S3 buckets for manifest uploads in dev clusters.

• Creating DynamoDB tables for feature-flag metadata in test clusters.

• Managing RDS instances for production-grade applications, per cluster.

---

Layer 3: Internal Architecture & Control Flow

System Components

1. Custom Resource Definitions (CRDs)

   • Define AWS services in YAML (e.g., kind: Bucket, kind: DBInstance).

   • Stored in Git and version-controlled per application/environment.

2. ACK Controllers

   • Watch for CRD creation/updates and perform reconciliation.

   • Use the AWS Go SDK and REST APIs to reflect desired state.

3. IAM Roles for Service Accounts (IRSA)

   • Each controller authenticates using an IAM role mapped to a Kubernetes service account.

   • Role boundaries are enforced at the environment or namespace level.

4. Argo CD + ApplicationSets

   • Detects changes to CRDs and deploys them across one or many clusters.

   • Provides status updates and rollback capabilities.

5. AccessPoint Security Gateway

   • Routes all outbound ACK API calls.

   • Logs, audits, and enforces token-based RBAC before allowing resource modification.

Control Flow in Orbit

1. A developer commits a new manifest (e.g., s3-bucket.yaml) to GitLab (C100).

2. Argo CD (C108) detects the change and applies it to the appropriate Orbit cluster.

3. The ACK S3 controller sees the CRD and issues an AWS API request to create the bucket.

4. AccessPoint (C52) validates and forwards the request.

5. Status is synced back to the CRD (status.conditions) and reflected in the Portal (C8).

---

Layer 4: Configuration Examples and Code-Level Implementation

Example: Provisioning an S3 Bucket

apiVersion: s3.services.k8s.aws/v1alpha1
kind: Bucket
metadata:
  name: airchina-app-storage
  namespace: airchina-dev-env
spec:
  name: airchina-app-storage-{{ .Environment }}

This file lives in the Orbit GitLab repo alongside application manifests.

Example: Argo CD ApplicationSet for ACK

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: k8or-aws-bucket-policy
spec:
  generators:
    - list:
        elements:
          - cluster: dev-airchina
            path: manifests/dev/aws
  template:
    metadata:
      name: '{{cluster}}-s3-bucket'
    spec:
      project: default
      source:
        repoURL: https://gitlab.k8or.com/app-repo.git
        targetRevision: HEAD
        path: '{{path}}'
      destination:
        server: '{{cluster}}'
        namespace: airchina-dev-env
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

IAM Role Mapping (IRSA)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:CreateBucket", "s3:PutBucketPolicy", "s3:DeleteBucket"],
      "Resource": "*"
    }
  ]
}

This policy is attached to the Kubernetes service account used by the ack-s3-controller.

---

✅ Benefits Recap

Benefit	Impact in k8or Orbit
Unified GitOps Management	Infrastructure and applications are deployed together.
Environment-Specific Provisioning	Dynamic per-cluster setups with no manual AWS ops.
Security via AccessPoint	Centralized RBAC + audit for all AWS provisioning.
Rapid Cluster Bootstrap	RDS, S3, and other infra can be provisioned in seconds.
Scalable to Multiple Clusters	ACK works across all dev/test/prod clusters via Argo.