Skip to main content

Multi-tenant Kubernetes workspaces: isolation, self-heal, and one config

How we built a Workspace operator that isolates each student in their own namespace, keeps platform-owned resources from drifting, and propagates one Git-managed template to every tenant — without requiring ArgoCD in the cluster.

JV

Joël Vappiani

16 min

Share on X

Kappsul started as a service offered at the University of Lorraine: students who needed an isolated environment to deploy an application could get one from a shared Kubernetes cluster, through a web interface. The platform had to turn a click into a namespace with a quota, access rules, a network boundary, and a place to run the student's app. That is a multi-tenancy problem before it is a Kubernetes problem: every student needs enough freedom to be useful, and a hard enough boundary that they cannot affect anyone else.

The first implementation used ArgoCD. Each workspace was an ArgoCD Application, all pointing at the same Git repository, and ArgoCD did the reconciliation. It was declarative, auditable, and it worked. But it made a central feature of our product depend on ArgoCD: creating a workspace required an ArgoCD installation and a Git source, and the platform could not run on a cluster that did not have them. We wanted the service to be portable to any Kubernetes cluster, and we did not want one of its core features to be delegated to an external tool.

We also realised we did not need all of ArgoCD. We needed two behaviours:

  • self-heal: a student must not be able to change the platform-owned parts of their workspace;
  • propagation: one change to the central configuration must update every workspace and its resources.

So we kept GitOps for what it is good at — changing the central configuration in Git — and wrote a small operator that belongs to the application. The operator creates the workspace envelope, keeps it from drifting, and propagates the shared template to every tenant. It uses only the Kubernetes API and a CRD, so it runs on any cluster.

This article is about that operator: the isolation model, the two GitOps behaviours, and the trade-offs.

Multi-tenancy is the reason the operator exists​

A workspace is not just a namespace. It is an isolation envelope:

LayerWhat it gives the studentWhat it protects the platform from
NamespaceA private place to deployName collisions and accidental cross-tenant access
ResourceQuotaA predictable CPU, memory, storage and pod budgetOne workspace starving the cluster
LimitRangeDefault requests and limits for containersPods without limits slipping through
RBACadmin inside their namespace, for the owner and guestsStudents acting outside their workspace
NetworkPolicyDefault-deny with explicit ingress and egressCross-namespace traffic and unexpected egress
Pod Security Admissionbaseline enforcement, restricted audit/warnPrivileged or unsafe pods

The operator's first job is to create that envelope for every student, in the same shape, with the right values. The student then gets admin inside the namespace so they can deploy and manage their applications. That is the multi-tenant contract: a private namespace with a hard boundary, and enough freedom inside it to be useful.

The boundary is enforced by Kubernetes primitives, not by the operator. The operator's job is to make sure the envelope exists and stays correct. That distinction matters for the rest of this article.

Portability: don't put an external tool on the critical path​

Portability, for us, means one thing: the operator must not need anything beyond a Kubernetes API and the CRD. No ArgoCD, no Git server, no network access to Git at runtime. That is the direct consequence of the first implementation: one ArgoCD Application per workspace, all pointing at the same Git repository. It worked, but it made workspace creation depend on an ArgoCD installation and its Git source. A core product operation was sitting behind a dependency that had nothing to do with the product.

The operator therefore uses only the Kubernetes API and a CRD. GitOps is still how the template reaches the cluster: the platform chart, including the workspace resource template, can be deployed by ArgoCD, Flux, Helm, or kubectl apply. The operator does not care. It is the runtime that turns that template into tenant namespaces.

That is the portability argument in one sentence: GitOps for delivery, an operator for runtime.

The two GitOps behaviours we actually needed​

We did not need all of ArgoCD. We needed self-heal and propagation, and nothing else from the GitOps toolbox.

  • Self-heal. If a student edits the quota, the limit range, the RBAC, or the network policy, the operator puts it back.
  • Propagation. The workspace envelope is defined once. Change that one configuration, and every workspace and its resources are updated to match.

We did not need a per-tenant Application, a sync dashboard per workspace, an app-of-apps, or multi-cluster orchestration. We needed a controller that owns the envelope, detects drift, and applies a shared template.

The operator also handles creation and deletion, because a tenant lifecycle needs both. But self-heal and propagation are the reason it exists.

The CRD is the tenant record​

The Workspace CRD is deliberately small. The spec is the input; the status is the operator's durable memory.

type WorkspaceSpec struct {
UserID string `json:"userId"`
UserEmail string `json:"userEmail,omitempty"`
UserCheckSum string `json:"userCheckSum,omitempty"` // generated by the controller
Guests []string `json:"guests,omitempty"`
Namespace string `json:"namespace"`
TemplateConfigMap string `json:"templateConfigMap,omitempty"`
}

type WorkspaceStatus struct {
Phase WorkspacePhase `json:"phase"`
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
ManagedResources []ManagedResource `json:"managedResources,omitempty"`
LastTemplateVersion string `json:"lastTemplateVersion,omitempty"`
PendingDeletions []PendingDeletion `json:"pendingDeletions,omitempty"`
}

managedResources is the list of resources the operator created, keyed by group, version, kind, namespace and name. lastTemplateVersion is the ConfigMap resourceVersion that was applied. pendingDeletions is the retry queue for resources that were removed from the template but not yet deleted.

Putting all of that in status is deliberate. The operator keeps no durable state of its own; a pod restart loses nothing. The CRD is the tenant record, and the namespace is the isolation boundary.

The loop​

The controller is a conventional informer-and-workqueue design, with three event sources:

  • a Workspace informer, scoped to the namespace that holds the CRs;
  • a ConfigMap informer, scoped to the namespace that holds the templates;
  • a dynamic resource watcher, watching everything labelled as owned by the controller, across all namespaces, for drift.

The queue is rate limited twice over: an exponential failure backoff from 5 milliseconds to 1000 seconds, and a token bucket at 50 items per second with a burst of 300. The exponential part protects a single broken workspace from hot-looping. The bucket protects the API server when a shared event fans out to everyone.

The most important optimisation is the one that does nothing. If a workspace is Active, its generation has not changed, there is no drift annotation, and its template version is current, the reconciler returns without a single API call. In steady state that skip path removes the vast majority of reconciliation work.

One template, many tenants​

The workspace envelope is defined once, in a ConfigMap. That ConfigMap is part of the platform chart, so it is managed in Git and delivered by whatever GitOps tool the cluster uses.

The template contains two layers of templating. The outer layer is Helm, evaluated when the platform is deployed. The inner layer is a Go template, evaluated by the operator for each workspace. The platform team controls the shape; the operator fills in the values from the CR.

# workspace-resource-template (rendered by Helm, then by the operator)
data:
namespace.yaml: |
apiVersion: v1
kind: Namespace
metadata:
name: {{ "{{" }} .Workspace.Namespace {{ "}}" }}
labels:
meta.kappsul.io/user-ns: "yes"
meta.kappsul.io/user-email: "{{ "{{" }} .Workspace.UserEmail | replace "@" "_at_" {{ "}}" }}"
meta.kappsul.io/user-checksum: "{{ "{{" }} .Workspace.UserCheckSum {{ "}}" }}"
pod-security.kubernetes.io/enforce: baseline

The same ConfigMap also defines the quota, the limit range, the RBAC bindings, and the network policy. That is the "one config" in the title: a single Git-managed object describes the envelope for every tenant.

The operator also uses the template for discovery. It does not hard-code a list of resource types to watch; it scans the template ConfigMaps, extracts the apiVersion and kind from each document, and registers a dynamic informer for the resulting GVRs. The set of watched kinds is derived at startup, so adding a new kind to the template requires a controller restart before its informer is registered.

Applying: server-side apply, in dependency order​

Rendering produces a map of YAML documents. The operator parses them into unstructured objects, sorts them by kind — Namespace first, then quota, limit range, service account, roles, role bindings, ConfigMaps, secrets, and network policies — and applies each one with server-side apply.

labels["app.kubernetes.io/managed-by"] = "workspace-controller"
labels["workspace.kappsul.io/name"] = workspace.Name
labels["workspace.kappsul.io/uid"] = string(workspace.UID)

// ...and then:

options := metav1.PatchOptions{
FieldManager: "workspace-controller",
Force: boolPtr(true),
}
_, err := resourceClient.Patch(ctx, resource.GetName(),
types.ApplyPatchType, data, options)

Server-side apply is the right primitive here for three reasons. It is idempotent, so a retry is always safe. It avoids read-modify-write races. And it records field ownership: the API server knows that workspace-controller owns these fields. The Force flag makes the operator win when someone else has touched a field it owns, which is exactly what self-heal needs.

The applier adds three labels and one annotation to every resource. The labels identify the owning workspace; the annotation stores a hash of the rendered spec and data. That hash is useful for bookkeeping and debugging, but it is not what drives drift detection. A hash can tell you that something changed. It cannot tell you who changed it.

Self-heal: who wrote this field?​

This is the first behaviour we needed from the GitOps toolbox.

The student has admin inside their namespace. That is intentional: they need to deploy applications, create services, manage their own resources. But the same permission lets them edit the quota, the limit range, the RBAC bindings, and the network policy. Those are platform-owned invariants, not student-owned resources.

The hard question is how to tell a student's edit apart from the operator's own write. A hash comparison tells you that the object no longer matches the last thing you wrote — but your own reconciliation changes it too, so the next event looks exactly like drift. And re-applying on every event makes it worse: your own writes trigger more events.

The answer is managedFields. Every write records a field manager. The drift watcher looks at the most recent entry and asks a simple question: was this workspace-controller or someone else?

lastModifier := getLastModifier(newObj.GetManagedFields())

if lastModifier == "workspace-controller" {
// Our own reconciliation. Not drift.
return
}

// Someone else changed a resource we own.
setDriftAnnotation(workspaceName)
enqueueWorkspace(workspaceName)

The reconciler sees the annotation, force-applies the desired state, and removes the annotation. The student's change disappears. The cycle is intentional and idempotent.

There is an important nuance here. Self-heal is a corrective control, not the security boundary. The security boundary is the namespace, RBAC, and NetworkPolicy: they decide what a student can reach. Self-heal protects the invariants inside the tenant's namespace, where the student is legitimately an admin. It is the difference between "you are allowed to edit this" and "this value is not yours to change".

It is also a heuristic, not a proof. An actor that writes with a manager name of workspace-controller would be indistinguishable from us. Our threat model is a student with kubectl, not a compromised controller identity.

Propagation: one config, every workspace​

This is the second behaviour we needed.

Changing the template is the highest-leverage operation in the system. We want it to be fast, visible, and rate-limited.

The ConfigMap informer stores the new resourceVersion in an in-memory cache and lists the workspaces from the informer cache — not from the API server. For each workspace that uses this template, it adds one item to the work queue. That is cheap. The queue, not the event handler, is the buffer.

func (c *WorkspaceController) handleConfigMapChange(obj interface{}) {
cm := obj.(*corev1.ConfigMap)
c.templateVersionCache.Store(cm.Name, cm.ResourceVersion)

workspaces, _ := c.workspacesLister.
Workspaces(c.config.WorkspacesNamespace).
List(labels.Everything())

for _, ws := range workspaces {
templateName := ws.Spec.TemplateConfigMap
if templateName == "" {
templateName = c.config.DefaultTemplateConfigMap
}
if templateName == cm.Name {
c.enqueueWorkspace(ws) // one queue item; no API call here
}
}
}

The reconciler then compares status.lastTemplateVersion with the cached version. If they differ, it reconciles and records the new version. At 50 workspaces per second, ten thousand workspaces converge in roughly three minutes.

This is the GitOps behaviour we wanted: one configuration in Git, propagated to every tenant. No per-tenant Application, no sync loop per workspace. The operator is the fan-out engine.

The lesson is not "fan out faster". It is "fan out into a queue". A loop that calls the API server ten thousand times is a denial-of-service attack you wrote yourself. A loop that enqueues ten thousand items and lets a rate-limited worker pool drain them is a rollout.

Lifecycle: create and delete​

Creation and deletion are plumbing around the two behaviours, but they still need care.

Creation. The operator adds a finalizer, generates a userCheckSum if it is missing, renders the template, applies the resources, and marks the workspace Active. The checksum and finalizer follow a "one step per pass" pattern: write, return, and let the update event bring the workspace back. Reconcile is a sequence of observations, not a single function call.

Deletion. There are two paths. Resources removed from the template are diffed against managedResources, added to status.pendingDeletions, and deleted asynchronously at a conservative five deletions per second with twenty retries and exponential backoff. The namespace itself is deleted with foreground propagation; the finalizer keeps the Workspace CR around until the namespace is gone.

The Namespace is explicitly excluded from the template diff. Deleting a namespace is the workspace-deletion path's job, not a template-diff side effect.

Protecting the control plane​

Everything above is a lot of writes. The design is shaped as much by what we refuse to do as by what we do.

BoundarySettingWhy
Informer resync30ssafety net for missed events
Reconcile queue50 QPS, burst 300fair progress during a template rollout
Deletion queue5 QPS, burst 10, 1 workera deletion storm is worse than a slow delete
Status updates50 QPS, batched every 30setcd writes are the expensive ones
Kubernetes client50 QPS, burst 300protect the API server from us
Deletion retries20, 5ms to 1000seventual success without hot loops

The operator runs as a single replica, and that is deliberate. It does not implement leader election. Running two replicas would mean two reconcilers writing the same status, two sets of duplicate API calls, and a class of conflicts that optimistic locking can only paper over. Instead we use a PodDisruptionBudget with maxUnavailable: 0 and a high priority class so that maintenance does not casually evict it. A pod restart costs ten to thirty seconds of control-plane work; the resync and the queue make that a non-event.

If you need true high availability, the answer is leader election with a coordination.k8s.io Lease, not more replicas behind a Service. We have not needed it yet. The operator is not on the real-time path, and the work is idempotent.

The bill​

This design is not free. Here is what we pay for it.

  • Self-heal is corrective, not preventive. A student can still make the change; the operator reverts it. If you need to prevent the write entirely, use RBAC and admission policies. Self-heal is the safety net for the resources the student is allowed to touch.
  • The template is a shared blast radius. A bad template change reaches every workspace at fifty per second. It is also one line in Git, which makes it look harmless. Treat a template change like a production rollout: stage it, watch the first workspaces, and have a rollback ready.
  • Two template layers are powerful and fiddly. The Helm escaping is ugly, and a missing pair of braces fails at render time, not at authoring time. The alternative — raw manifests baked into the operator image — loses the GitOps boundary. We chose the flexibility and accepted the syntax.
  • Drift detection trusts managedFields. It is a very good signal, but it is a signal, not an authorization system. An actor that can write with our field manager name can evade it.
  • You own the operator. That is the portability trade-off. You no longer depend on ArgoCD in the cluster, but you now maintain the controller, its CRD, and its failure modes.
  • There is no per-tenant Git audit trail. The CR status is the state, not a commit per workspace. If you need an audit trail of every tenant change, you need to add one.

Takeaways​

  • Start with the isolation model. A workspace is a namespace plus quota, limit range, RBAC, network policy, and pod security. The operator exists to make that envelope consistent.
  • Keep the runtime portable. Do not make an external tool a runtime dependency for a core product operation. GitOps for delivery, an operator for runtime.
  • Take only the GitOps behaviours you need. Self-heal and propagation were enough. You do not need a full sync engine to get them.
  • Use server-side apply. Idempotent writes, no read-modify-write races, and field ownership that means something.
  • Detect drift by who, not what. managedFields tells you whether the change came from you or from someone else; a hash cannot.
  • Fan out into a queue, not into API calls. The queue is the buffer that turns a large template change into a rollout.
  • Separate the security boundary from the corrective control. RBAC and NetworkPolicy decide what a student can reach. Self-heal protects the invariants inside their namespace.