k8or ISO: Unified Linux, K8s, and Image Cache – Four-Layer Documentation

By Raj Marni. April 20, 2025. Revised. Version: 0.0.04

📘 Layer 1: The Big Picture – High-Level Architectural Overview

🔹 Introduction

The k8or ISO: Immutable Linux + K8s + Image Cache is a unified bootable image that bundles the operating system, Kubernetes control plane, and a pre-baked container image cache into a single, immutable layer. It is purpose-built to support fast, repeatable, and secure Kubernetes deployments, especially in air-gapped, low-connectivity, or scale-out environments.

This ISO is not a traditional Linux installer. Instead, it delivers a production-ready, bootable system image powered by Talos Linux, where Kubernetes is tightly integrated into the OS layer, and container images are available immediately from local disk—eliminating the need to fetch them over the network post-boot.

🧭 Architectural Purpose

At its core, the k8or ISO simplifies K8s provisioning by eliminating the conventional multi-step process of:

  1. Installing a base OS

  2. Installing K8s on top of it

  3. Customizing above enviroment by pulling container images from external registries

Instead, it delivers a single-purpose built bootable artifact that includes:

  • An immutable, security-hardened Linux base (Talos)

  • Upstream Kubernetes components

  • An optional pre-populated image cache for workload and dependency containers

🧱 Architectural Layers in k8or ISO

Layer
Description

Immutable Linux OS

Based on Talos Linux, a stripped-down Linux OS purpose-built for K8s. No shell, SSH, or mutable filesystem.

Kubernetes Runtime

Bundles upstream kubelet, etcd, kube-apiserver, and containerd. Fully managed by Talos APIs.

Image Cache

A preloaded local OCI image store mounted at boot and accessed by Talos’ internal registry. Acts as a pull-through cache for containerd.

🛰️ Orbit Plane & Cluster Plane (Contextual Placement)

Within the broader k8or Orbit ecosystem, this ISO plays a vital role at the Cluster Plane level. It is typically provisioned by the Orbit Plane through:

  • PXE booting (Mender), virtual media, or USB flashing

  • Automated deployment pipelines in air-gapped factories or datacenters

Once booted and configured, the node joins the Kubernetes cluster—ready to run workloads without requiring an internet connection or external image registry.

🎯 Key Design Goals

  • Zero-dependency bootstrapping – no need for live internet or provisioning scripts

  • Fast startup times – images are ready on-disk at boot

  • Immutable security – unmodifiable OS surface area prevents drift or tampering

  • Deterministic deployments – ensures the same image and Kubernetes configuration every time

  • Air-gapped support – ideal for environments without outbound network access

📈 Business & Operational Value

Benefit
Description

Operational Speed

Rapid provisioning across hundreds or thousands of nodes

Resilience

No reliance on external registries or package managers

Security

Locked-down OS, reduced attack surface, audit-friendly

Simplicity

One image = one environment = fewer provisioning errors

Scalability

Ideal for fleet management, factory-floor rollouts, and remote edge devices

📘 Layer 2: Individual Component – k8or ISO: Immutable Linux + K8s + Image Cache

🧩 Component Name

k8or ISO (Backed by Talos Linux + Kubernetes Runtime + Image Cache)

🎯 Core Purpose & Functionality

The k8or ISO component provides a bootable, self-contained environment that encapsulates:

  1. A minimal, immutable Linux OS (Talos)

  2. A complete Kubernetes runtime

  3. An optional image cache for bootstrapping workloads without remote image pulls

It is designed to:

  • Enable cluster provisioning in air-gapped environments

  • Accelerate Kubernetes node spin-up

  • Ensure deterministic, secure, and reproducible deployments

This ISO serves as the entry point to the Cluster Plane in the k8or Orbit architecture.

🔗 Key External Interactions

Target System / Component
Interaction Type
Purpose

Orbit Plane

Deployment Trigger

Initiates ISO provisioning to bare metal or VM nodes

PXE Boot / USB / VM Host

Boot Interface

The ISO is delivered via PXE, virtual media, or USB flash

Talosctl

API Client

Talos configuration and Kubernetes bootstrap automation

Image Cache RegistryD

Local Pull-through Registry

Containerd accesses pre-baked container images from this cache

k8or GitOps Controller

Optional

Post-boot, cluster connects to Orbit for GitOps-driven workload sync

🔄 High-Level Data & Control Flows

  1. Boot Sequence

    • Node boots from ISO → loads Talos kernel → enters maintenance or install mode

  2. Configuration Application

    • talosctl applies machine config (role, disk, image cache flag, etc.)

  3. Installation to Disk (Optional)

    • If specified, Talos installs itself and image cache to system disk

  4. Kubernetes Bootstrap

    • talosctl bootstrap initializes control plane or joins worker to cluster

  5. Image Fetching

    • containerd pulls required images from embedded image cache via registryd

    • If an image isn’t cached, and networking exists, it falls back to upstream

📦 Component Dependencies

Dependency
Purpose

Talos Linux

Secure, immutable OS foundation for Kubernetes

Talosctl CLI

Interface for applying configuration, bootstrapping

OCI Image Registry

Optional source for building image cache pre-ISO

Docker/Podman

Required to generate the ISO with image cache via imager container

Local Storage (ISO or Disk)

Required to store OS + Kubernetes + cache in one place

🌐 Deployment Context

This ISO is typically used in the following deployment modes:

Mode
Description

Air-gapped

No internet connectivity; relies fully on preloaded image cache

High-scale

Mass provisioning across clusters or edge devices using consistent image

Hybrid

Has fallback internet access; uses cache as first source, remote as backup

✅ Benefits Delivered by This Component

  • Plug-and-Play Provisioning: Zero network dependency for Kubernetes + workloads

  • Speed: Faster boot and ready state due to pre-baked images

  • Security: Immutable OS with no shell, SSH, or mutable userland

  • Control: You decide what images go into the system; no runtime pulls unless explicitly allowed

  • Consistency: Same boot environment across all target nodes

📘 Layer 3: Internal Architecture Deep Dive – k8or ISO

🧩 Internal Modules & Subsystems

The k8or ISO is internally composed of tightly integrated components, bundled during ISO creation and orchestrated at runtime by Talos Linux.

Module
Description

Talos Boot Loader

Initializes immutable Linux environment and transitions into kernel space

containerd

Primary container runtime used by Kubernetes and other internal services

registryd

Lightweight OCI-compatible pull-through cache that exposes cached images to containerd

image cache mount

Filesystem-mapped OCI layout directory prepopulated during ISO creation

kubelet

Kubernetes node agent (control plane or worker) managing pod lifecycle

Talos API Server

Receives all control plane instructions (e.g., config, patches) via talosctl

kube-apiserver, etcd, controller-manager, scheduler

Launched on control plane nodes after bootstrap, managed by Talos runtime

🔁 Data Flow: Boot to Workload-Ready State

[ISO Boot]
   ↓
[Talos Kernel Init]
   ↓
[Filesystem mounts image cache to /var/lib/registry]
   ↓
[registryd starts and exposes OCI cache as pull-through registry]
   ↓
[containerd requests images via registryd]
   ↓
→ [Image found in local cache] → Served instantly from disk  
→ [Image missing in cache] → Fallback to upstream (if allowed)

📥 ISO Creation Workflow (Build-Time Internals)

  1. Image List Definition

    • Developer defines list via images.txt (e.g., defaults + custom CNI, app images)

  2. Image Cache Generation

    • talosctl image cache create builds a folder with OCI-compliant blobs and manifest files

  3. ISO Construction

    • Run Talos-provided containerized imager tool:

      docker run -v ./cache:/cache -v ./out:/out ghcr.io/siderolabs/imager:1.9 \
  iso --image-cache /cache --output /out/k8or.iso

  4. ISO Contents

    • Talos boot files

    • Configuration injection support

    • Pre-mounted image cache in OCI layout format

🔐 Protocols, Security & Access Control

Subsystem
Protocols / Standards Used

Talos API

gRPC (over mutual TLS)

Container Runtime

containerd with snapshotter (default overlayfs)

Image Cache Access

Local HTTP registryd over Unix socket

Registry Format

OCI image layout + SHA256 blob hashing

Kubernetes Bootstrap

Certificate-based trust and etcd quorum formation

Talos ensures that registry access is locked down, only visible to local containerd. No internet or remote shells are exposed by default.

⚙️ Configuration Injection & Overrides

Talos supports injecting configuration patches to:

  • Enable image cache (machine.features.imageCache.local: true)

  • Define volumes for cache mount

  • Set install disk and boot parameters

This is done using talosctl apply-config or via PXE/USB configuration injection.

📋 Error Handling & Observability

Area
Logging/Behavior

registryd

Logs all image fetches (hits and misses)

image cache status

talosctl get imagecacheconfig shows status, source (ISO or disk), and copy readiness

bootstrap stage

Fails if core images (e.g., etcd, kube-apiserver) are missing from cache and unreachable

fallback behavior

On cache miss, attempts to reach upstream registry (if allowed) and logs delays/errors

🔍 Example Internal View – Local Registry Pull

  • containerd requests: docker.io/coredns/coredns:1.9.3

  • registryd intercepts request via pull-through logic

  • If sha256 blob exists in mounted cache:

    • Immediately returns image layer

  • If not:

    • Attempts external fetch (only if network is allowed)

    • Logs a MISS event

📘 Layer 4: Code-Level Explanations – k8or ISO

📁 Repository & Tools Overview

Tool / Repo
Description

talosctl

CLI tool to interact with Talos nodes: image cache creation, config application, bootstrap

imager

Talos-supplied Docker container to build custom ISO or raw disk images

images.txt

Text file listing all required OCI images to be baked into the cache

machineconfig.yaml

Talos MachineConfig file, defines node type, install disk, features (e.g. image cache)

imagecache_patch.yaml

YAML patch to enable image cache support at runtime

🏗️ Build Process: End-to-End Steps with Code

1. Generate Default Image List

talosctl images defaults > images.txt

Edit the file to append additional images like custom CNIs, storage drivers, or app base images:

docker.io/calico/node:v3.25.1
docker.io/myregistry/ai-infer-service:v2.0.0

2. Create the OCI-Compatible Image Cache

cat images.txt | talosctl image cache create --output ./image-cache

This produces:

  • oci-layout

  • blobs/sha256/... (image layers)

  • index.json with manifest

3. Build the K8or ISO with Preloaded Cache

docker run --rm -it \
  -v "$(pwd)/image-cache:/cache" \
  -v "$(pwd)/_out:/out" \
  ghcr.io/siderolabs/imager:1.9 \
  iso --image-cache /cache --output /out/k8or.iso

This step uses Talos' official imager to bundle Talos kernel, rootfs, and image cache into a bootable ISO.

4. Apply Configuration via talosctl

Boot the target machine from the ISO. Then apply your configuration:

talosctl apply-config \
  --insecure \
  --nodes <NODE_IP> \
  --file machineconfig.yaml

Example machineconfig.yaml (Control Plane)

machine:
  type: controlplane
  install:
    disk: /dev/nvme0n1
  features:
    imageCache:
      local: true
cluster:
  name: k8or-cluster
  controlPlane:
    endpoint: https://192.168.1.10:6443

5. Optional Patch for Volume-Based Cache

# imagecache_patch.yaml
- op: add
  path: /machine/disks/0/partitions
  value:
    - mountpoint: /var/lib/registry
      name: image-cache
      label: image-cache

Apply patch with:

talosctl apply-config \
  --insecure \
  --nodes <NODE_IP> \
  --file imagecache_patch.yaml

🔍 Observing & Debugging at Runtime

View image cache status:

talosctl get imagecacheconfig

Monitor registry cache behavior:

talosctl logs registryd

Typical logs:

[INFO] GET /v2/docker.io/library/coredns/blobs/sha256:...

🧪 Testing & Validation

Area
Test Action

ISO Integrity

Boot ISO on real or virtual hardware (QEMU/Proxmox/ESXi)

Image Availability

Deploy test workload that uses pre-baked images

Registry Miss

Try pulling an image not in cache and validate fallback behavior

Performance

Measure boot-to-ready time with vs. without cache

🧼 Clean Build Script Example

#!/bin/bash
set -e

# Step 1: Generate image cache
cat images.txt | talosctl image cache create --output ./image-cache

# Step 2: Build ISO
docker run --rm -it \
  -v "$(pwd)/image-cache:/cache" \
  -v "$(pwd)/_out:/out" \
  ghcr.io/siderolabs/imager:1.9 \
  iso --image-cache /cache --output /out/k8or.iso

📦 Deployment Integration

Stage
Tool
Description

USB Flash

dd if=k8or.iso of=/dev/sdX bs=4M status=progress

Use for edge device provisioning

Virtual

PXE or Virtual CD

Mount ISO directly on cloud VM or hypervisor

Fleet Automation

GitOps or custom PXE boot service

ISO can be served with injected configs per node or role

PreviousOrbit High-Level Architecture DiagramNextStreamlining Docker to K8s Deployments with the k8or Orbit and Cluster Plane

Last updated