Component C56 – SyncMaster
By Raj Marni. March 26, 2025. Revised. Version: 0.0.01
1. Overview
C56 – SyncMaster is responsible for ensuring data consistency, state synchronization, and event propagation across the k8or Orbit ecosystem. As a meta-component managing synchronization between different parts of the system (e.g., k8or Portal, Helm chart storage, Docker image build process), SyncMaster facilitates real-time updates and ensures that all components are in sync. It acts as a central hub for communication between components, handling data flow, event handling, and policy enforcement across the system.
2. Internal Architecture
2.1 Core Modules
Event Listener & Publisher
Function: Listens for changes or updates in the system and publishes them to relevant components.
Process:
Acts as a subscriber to events generated by other components (e.g., new Helm charts generated, Docker images built).
Uses message queues (NATS) to listen for changes and then publishes those events to the appropriate subscribers.
Implementation Detail:
The event listener is implement event-driven architectures using technologies REST APIs to propagate updates to connected components.
Data Synchronization Engine
Function: Coordinates the synchronization of data and configurations across the system, ensuring consistency across distributed components.
Process:
When data changes in one component (e.g., C8 – Portal, C12 – Manifestor, C20 – Chart Store), the synchronization engine triggers updates in other components that depend on this data.
Manages the state replication across multi-cluster setups or between distributed systems.
Implementation Detail:
SyncMaster uses distributed data stores or event sourcing patterns to ensure that the data is replicated across all required components without conflicts or delays.
Policy Enforcement & Integrity Checker
Function: Ensures that changes made to data or configurations meet the predefined policies, ensuring integrity and compliance.
Process:
Checks each incoming event or data update to ensure it adheres to security, data consistency, and operational policies.
If data is inconsistent or violates security policies (e.g., invalid metadata, unauthorized access), SyncMaster raises alerts or halts the synchronization process.
Implementation Detail:
Uses RBAC (Role-Based Access Control) and IAM (Identity and Access Management) systems to validate the integrity of incoming changes, ensuring that only authorized components or users can modify certain aspects of the system.
Communication Hub (API Gateway)
Function: Provides a unified interface for components to communicate and trigger synchronization processes.
Process:
Exposes a set of RESTful APIs endpoints** for other components (like C8 – K8or Portal or C12 – Manifestor) to trigger synchronization operations or request updates.
The API gateway can serve as an interface for external systems to send synchronization requests, ensuring that multi-cluster or multi-region communication is standardized.
Implementation Detail:
Typically built on scalable, high-performance API management solutions (e.g., Kong, API Gateway) for fast processing of requests across components.
Audit Logging & Monitoring
Function: Records all synchronization events, changes, and updates for auditing and monitoring purposes.
Process:
Captures the flow of data, logging every state change, synchronization event, or data modification in real-time.
This logging allows administrators to monitor system health, track issues, and review historical changes.
Implementation Detail:
Uses a time-series database (e.g., Prometheus or ElasticSearch) to store logs and metrics related to synchronization events, enabling real-time analysis and alerting.
3. Data Flows & Communication Patterns
3.1 Event-Driven Synchronization
Event Generation:
An event is generated when any relevant data changes occur (e.g., an image is built in C4, or a Helm chart is generated in C24). These events are published to a message broker or event bus (NATS).
Event Handling by SyncMaster:
SyncMaster subscribes to these events and processes them by invoking the Data Synchronization Engine to propagate the changes across other components.
The API Gateway in SyncMaster can also expose endpoints for manual synchronization triggers from components like C8 – Portal or C12 – Manifestor.
Policy Check and Integrity Validation:
Before propagating changes, SyncMaster checks each event against pre-configured security policies (e.g., ensuring that only authorized changes are made).
The Policy Enforcement module validates the integrity of the data and ensures compliance with operational standards.
Data Propagation & Replication:
Once validated, SyncMaster synchronizes data and configurations to dependent components (e.g., C20 – Chart Store, C8 – Portal, C28 – Kustomize).
In multi-cluster setups, SyncMaster ensures that changes are replicated across multiple regions or clusters to maintain consistency.
Audit Logging & Monitoring:
Every event and change made by SyncMaster is logged for future reference and troubleshooting.
Logs and metrics are aggregated and pushed to centralized monitoring systems, Prometheus and Grafana in meta-component InsightHub , to ensure system health and performance.
3.2 Interactions with Other Components
From C8 – K8or Portal
SyncMaster interacts with the K8or Portal (C8) to propagate updates about user interactions (e.g., environment configurations, image uploads).
Process: The Portal sends configuration or image metadata to SyncMaster, which then synchronizes the data across the relevant components like C12 – Manifestor or C20 – Chart Store.
To C12 – Manifestor
SyncMaster ensures that the Manifestor is updated with the latest metadata or configuration updates when there are changes triggered by other parts of the system.
Process: SyncMaster synchronizes the image data, metadata, or user-defined configuration for Helm chart generation in C12.
To C20 – Chart Store
C20 – Chart Store holds versioned Helm charts, and SyncMaster ensures that any changes to these charts are reflected in the centralized Chart Store repository.
Process: After applying the final configuration to the Helm charts, SyncMaster pushes the new charts to C20 for storage and access.
To C28 – Kubernetes Kustomize
SyncMaster interacts with C28 (Kustomize) by passing over the environment-specific Helm charts, ensuring that Kustomize has the correct data for further customization.
Process: C28 customizes the Helm charts, and SyncMaster ensures that these changes are properly synchronized back to the Chart Store or other storage solutions.
4. Technology & Protocols
Event Streaming & Messaging:
NATSfor event-driven architectures and message passing between SyncMaster and other components.
RESTful APIs & gRPC:
SyncMaster uses RESTful APIs to facilitate communication between components and manage synchronization tasks in real-time.
IAM & RBAC:
Role-Based Access Control (RBAC) integrated for policy enforcement on data and configuration changes.
5. Benefits & Impact
Real-Time Synchronization
Ensures that all parts of the system remain in sync with the latest configuration, environment settings, and metadata.
Event-Driven Architecture
The event-driven nature allows the system to react dynamically to changes without requiring continuous polling or manual updates.
Improved System Reliability
With data integrity checks and policy enforcement, SyncMaster ensures that changes are always compliant and safe to propagate.
High Availability & Scalability
By replicating data and configuration changes across clusters and regions, SyncMaster ensures high availability and fault tolerance in distributed environments.
Centralized Control
SyncMaster centralizes data synchronization and event propagation, reducing complexity and ensuring that updates are efficiently managed across multiple components.
Last updated