Kappsul is a platform we built at the University of Lorraine: from a web interface, any student can get a private, isolated environment on a shared Kubernetes cluster and deploy their own applications in it. This is the story of the day we switched how those environments are managed — a migration that ran green from start to finish, and deleted every workspace created after the first two hundred, along with everything inside them.
It is also a story with a happy ending, and the ending has a name: Velero.
If you have read Multi-tenant Kubernetes workspaces: isolation, self-heal, and one config, you know the destination of this migration and why we chose it. This post is the part of the story that article deliberately skips: the day we switched over, and the two hours during which everything went wrong. It is written to be read on its own.
What Kappsul is, and why isolation is the product
Universities have a specific Kubernetes problem. Courses and student projects need somewhere to run an application — a web app, an API, a database — but you cannot give every student their own cluster, and you cannot give untrusted users access to a shared one. So the platform does something in between: a student clicks a button, and gets a namespace on the shared cluster, with everything configured so they can work safely inside it.
That sounds trivial. It is the whole product. The hard part is not creating a namespace —
kubectl create namespace is one command. The hard part is the contract:
A private namespace with a hard boundary, and enough freedom inside it to be useful.
"Enough freedom" means the student is admin inside their namespace: they can deploy
with Helm, write manifests, expose services, manage their own secrets. "Hard boundary"
means none of that freedom can leak out — onto other students, or onto the platform
itself. In practice, every workspace is an envelope of Kubernetes primitives:
| Layer | What the student gets | What the platform is protected from |
|---|---|---|
| Namespace | A private place to deploy | Name collisions, cross-tenant access |
| ResourceQuota | A predictable CPU/memory/storage budget | One workspace starving the whole cluster |
| LimitRange | Default requests and limits on containers | Pods without limits slipping through |
| RBAC | admin inside their own namespace only | Students acting outside their workspace |
| NetworkPolicy | Default-deny, explicit ingress/egress | Cross-namespace traffic, unexpected egress |
| Pod Security Admission | baseline enforcement | Privileged or unsafe pods |
One more thing matters for this story: the tenants are students. They are learning Kubernetes. They will delete things, edit things, break things — sometimes on purpose, just to see what happens. The platform cannot prevent every mistake inside the workspace; it has to put the workspace back when it happens. Remember that: it is why self-healing was a requirement, and it is how a deletion tool ended up holding a loaded weapon.
The ArgoCD era: a good tool, used well
For the first year, the lifecycle of every workspace was managed by ArgoCD. Creating
a workspace meant creating an ArgoCD Application — one per workspace — pointing at a
Git repository that contained a small Helm chart: the envelope from the table above.
ArgoCD watched Git, rendered the chart, and applied the resources into the student's
namespace. Change the chart in Git, and every workspace converged to the new state.
I want to be fair to this design, because it was good:
- Declarative and auditable. Every workspace's desired state lived in Git.
- Self-healing for free. ArgoCD's whole job is noticing drift and correcting it — a student edits the quota, ArgoCD puts it back. We got our hardest requirement from a tool we didn't have to write.
- Battle-tested. Reconciliation, retries, sync semantics: solved problems, not our problems.
- Zero custom controller code. Nothing to maintain, nothing to debug at 2 a.m.
So why did we leave? Four reasons, in increasing order of importance:
- It sat on the critical path. Creating a workspace — the most basic operation of the product — required ArgoCD and GitLab to be up. When either had an outage, students could not get workspaces at all. A dependency that had nothing to do with the product could take the product down.
- It killed portability. The platform could not run on a cluster without ArgoCD and a Git server. We wanted Kappsul to run on any Kubernetes cluster.
- It was overkill. We were using a full delivery platform to reconcile six resources per tenant. No sync dashboards per workspace, no app-of-apps, no multi-cluster — none of that was ever going to be used.
- Scale mismatch. Thousands of tiny Applications, each refreshed forever by the ArgoCD controller, for what was ultimately a namespace with four policy objects in it.
The destination: one CRD, one operator, two behaviors
The replacement is deliberately small. A Workspace custom resource is the tenant
record — who owns it, which namespace, which guests. A small operator watches those CRs
and does exactly two things, which turn out to be the only two things we ever needed
from the GitOps toolbox:
- Self-heal: if a student edits a platform-owned resource, put it back — detected
not by comparing hashes, but by asking
managedFieldswho made the last write; - Propagation: the envelope is defined once, in a Git-managed template ConfigMap; change it once, and the operator fans it out to every workspace.
GitOps stays for delivery — the template lives in Git and reaches the cluster by whatever tool the cluster uses. The runtime needs nothing but the Kubernetes API and the CRD. That is the whole argument: GitOps for delivery, an operator for runtime — and it is covered in depth in the operator article.
What the diagram does not show is the part that matters for the rest of this story: deletion. When a Workspace CR is deleted, the operator deletes the student's namespace — foreground propagation, finalizer held until the namespace is gone. The operator owns the namespace lifecycle now. Remember that too.
The plan: a migration job
So: several thousand live workspaces, each currently an ArgoCD Application, had to become Workspace CRs. No downtime, no data loss, no "we'll fix it as we go". The operator was ready; the switch was the problem.
Two properties of the old world shaped the whole design, and you need both to understand the failure:
Fact 1 — the cascade finalizer. Kubernetes finalizers are strings on an object that
mean "do not actually delete me until the controller that owns this string says so".
Every workspace Application carried
resources-finalizer.argocd.argoproj.io, which is ArgoCD's promise: when this
Application is deleted, delete everything it ever created — the quota, the RBAC, the
network policies, and the namespace itself. Deleting a namespace deletes everything
inside it: every student Deployment, Service, ConfigMap, Secret, and every PVC with its
data. That finalizer is why the migration could not just delete the Applications and
hope.
Fact 2 — the Applications were alive. ArgoCD's application controller refreshes
every Application on its reconciliation cycle, rewriting status fields each pass. Every
one of those writes bumps the object's resourceVersion — the concurrency token that
Kubernetes uses to reject writes based on stale data.
The migration job's logic, per Application:
- Convert the ArgoCD Application into a
WorkspaceCR (same name, same target namespace, user and guests extracted from the Helm values); - Create the Workspace CR and wait for the operator to reconcile it — poll
until the workspace reports
Activewith all envelope resources created; - Strip the cascade finalizer from the Application, so that deleting it deletes only the Application object;
- Delete the ArgoCD Application.
The student's own deployments, services and volumes were never ArgoCD's business — they were installed by our application service through Helm, directly in the namespace. On paper, the handover was clean: the operator takes over the envelope, the student's stuff was never ArgoCD's to delete.
And the job itself was genuinely careful. It processed Applications in batches with delays between them, rate-limited its API calls, tracked progress in a file on a persistent volume so it could resume, supported a dry-run mode, and refused to delete an Application before its Workspace CR was fully reconciled. We were proud of it. That is not sarcasm — every one of those decisions was correct. The bug was elsewhere.
The rehearsal
We tested the job in preprod first. Preprod had a handful of workspaces.
The run took minutes. Every Application was converted, every Workspace CR reconciled almost instantly, every finalizer stripped, every Application deleted. We compared namespaces before and after: identical, envelope and all. We let the operator run for a while, checked the workspaces were self-healing, and called it done.
It was done — for a platform the size of preprod. What we had actually tested was the code. What we had not tested was the clock. Hold that thought.
Production, Thursday
In early December 2025, we ran the migration against production. The configuration was
the one designed for our scale: all workspaces (WORKSPACES=*), deletion enabled
(DELETE_ARGOCD=true), batches of 25 Applications with a 180-second pause between
batches, five concurrent migrations, and a 15-minute timeout for the operator to
reconcile each Workspace.
The run started green. The first Applications converted, reconciled, and deleted exactly like in preprod. The job churned through batch after batch, logging a steady stream of checkmarks.
The first two hundred or so workspaces migrated perfectly.
Then, without anything changing — no error, no restart, no threshold crossed — the workspaces behind the checkmarks started to die.
The race we didn't know we were running
Here is the mechanism, and it is embarrassingly simple once you see it.
The job fetched all Applications in a single LIST at the very beginning of the
run. From that moment on, every Application object it held in memory started aging.
Meanwhile, ArgoCD's application controller kept doing its job, refreshing every
Application on its cycle and bumping each object's resourceVersion as it rewrote
status.
For each Application, the job then:
- created the Workspace CR,
- waited — up to fifteen minutes — for the operator to reconcile it,
- and only then tried to strip the cascade finalizer, using a full-object
UPDATEof the object it had been holding since the start of the run.
A full-object UPDATE is a read-modify-write with optimistic concurrency: the write
carries the resourceVersion you read, and if the live object has moved on since, the
API server rejects it with a 409 Conflict. By the time the job reached an Application —
after its Workspace CR had been created and reconciled, after the batches and the delays
— the object it was holding was minutes to hours old. ArgoCD had rewritten it many
times over.
// the strip step, reduced to its essence
finalizers, _, _ := unstructured.NestedStringSlice(app.Object, "metadata", "finalizers")
// ...remove the ArgoCD finalizer from the in-memory copy...
_, err := dynamicClient.Resource(applicationsGVR).
Namespace(app.GetNamespace()).
Update(ctx, app, metav1.UpdateOptions{}) // full-object write of a stale object
For the first Applications, processed while the run was young, the in-memory object was still fresh enough that the write won the race. The finalizer was stripped, the Application was deleted cleanly, and the operator took over a fully intact namespace.
For every Application after that, the write lost the race. The API server rejected it. The finalizer survived. And the job — which treated a failed strip as a reason to log and move on, not to stop — deleted the Application anyway.
And the cascade finalizer was a promise. ArgoCD deleted every resource that Application had ever created — the ResourceQuota, the LimitRange, the NetworkPolicies, the RBAC bindings, and the namespace itself. Deleting a namespace deletes everything in it: every student Deployment, StatefulSet, Service, ConfigMap, Secret, and every PVC with its data.
The staleness curve explains the boundary we observed. Batch k ran at roughly
T0 + k × (reconcile wait + batch delay). The probability that an Application had been
touched by ArgoCD since T0 climbs from "possible" to "certain" over the first couple
of hours. Two hundred workspaces in, the job had crossed from "usually wins the race" to
"always loses it". Nothing in the code changed at that point. Only the clock moved.
The silence
Here is the part I still find the most unsettling: nothing failed.
The migration job reported success, because from where it stood, everything had succeeded. The Workspace CR was created — true. The Application was deleted — also true. The job finished, printed its summary, and exited zero.
ArgoCD reported nothing, because deleting the resources behind an Application with the cascade finalizer is not an error — it is the contract. ArgoCD was not even misbehaving that day: somewhere in the cluster, a controller was faithfully honoring a promise we had made a year earlier and forgotten.
Our operator reported nothing either. Its reconciliation loop saw namespaces that no longer existed, and did exactly what it is designed to do: it recreated the envelope — namespace, quota, limit range, network policies, RBAC — with server-side apply. The platform looked alive. Workspaces were up, reachable, and hollow.
And our logs? The run produced no warnings that reached anyone. The one component that could have flagged the failure mode — the strip step — was written to survive its own guard failing: log, and continue. A guard that does not stop the run is not a guard; it is a disclaimer. The post-mortem found it in the code within minutes. Finding it in the logs, after the fact, among a million green lines, was another matter.
The first reports came from users: workspaces that were up, reachable, and empty.
The weekend
This is the part of the story where I get to be grateful instead of clever.
We had Velero backups. Of everything — Kubernetes objects and volume data. Not because we are prophets, but because "the cluster eats a namespace" had been on our risk list since day one, and Velero was the answer we had set up and then mostly stopped thinking about.
The team spent the weekend restoring. Every workspace namespace, every object, every PVC and the data inside it, back to the state they were in before the run. It was long, unglamorous work: checking restore order, reconciling what belonged to what, verifying that each workspace came back whole. But it was possible, and by Monday the platform was back, and the incident became a story instead of an ending.
Two things made the difference:
- The backups covered the right things. Kubernetes objects and persistent volume data. A backup of one without the other would have restored the letter and lost the spirit.
- Nobody had to improvise. The restore procedure existed, the tool was installed, the backups were there. The weekend was spent executing, not designing.
The fix we almost didn't need
For completeness: the job did get fixed. The strip step was replaced by a delete that asks Kubernetes to orphan dependents, and — more importantly — the job learned to count. Before deleting an Application it counted the Deployments, StatefulSets and Services in the target namespace; after the delete, it counted again, and hard-failed the entry if the number dropped. A canary per namespace, cheap and brutal.
It did not matter much, because by then there was nothing left to migrate: the cascade had done the conversion in the most literal sense. Every workspace was managed by the operator now. That is the darkest line in this story, and the truest: the migration succeeded. We had just planned for it to complete by conversion, not by deletion.
A few days later we deleted the migration job from the repository entirely. New workspaces are born as Workspace CRs. Nobody will ever run that job again, and the next platform will never have to.
What we learned
1. Scale is a test dimension. We tested the job on a handful of workspaces and it passed, because it was correct for a handful of workspaces. The failure mode only exists at runtimes longer than the freshness window of the objects being migrated. The fix to our test strategy: ramp deliberately — 10, then 100, then 1,000, then all — and treat a passing run at one scale as evidence about that scale only.
2. Time is a test dimension. Preprod was green because it was small: the whole run fit inside the window where nothing had gone stale. Production ran for hours, and the bug lives in the hours. Testing in real conditions does not mean testing on production-sized data — it means testing with production-shaped durations, or accepting that a time-dependent bug is invisible to a short run.
3. Never read-modify-write stale state on a destructive path. The strip step held an object for minutes-to-hours and then wrote the whole thing back. The correct patterns are boring and well-known: patch the single field you care about, or re-read the object and retry on conflict, or use server-side apply with a dedicated field manager. We knew these. The operator itself is built on server-side apply precisely because it avoids read-modify-write races. The migration job was not held to the same standard as the thing it was migrating to. That asymmetry is the real root cause.
4. A guard that doesn't stop the run is not a guard. The finalizer strip could fail,
log, and let the deletion proceed. On a path that ends in DELETE, every failed
precondition must abort — loudly, and ideally at the batch level, not the item level.
"Warn and continue" is a reasonable default for retries and a catastrophic one for
irreversible steps.
5. Measure outcomes, not steps. The job verified its own actions: CR created, Application deleted. Both true. Neither was the thing that mattered. The fixed job counted resources in the namespace before and after — a canary that would have turned the silent cascade into a hard failure within the first batch. Ten lines of code, had it existed then.
6. Finalizers are loaded weapons.
resources-finalizer.argocd.argoproj.io is not an implementation detail; it is a
standing order to destroy everything the object ever created, namespace included. Before
any migration that deletes objects, audit what each finalizer will cascade to. We had
written that order ourselves, a year earlier, and forgotten it was still standing.
7. The most dangerous failure is every component working as designed. The job succeeded. ArgoCD honored its contract. The operator healed the envelopes. Nobody lied, nobody crashed, nothing alerted — and several thousand workspaces were gone. When every layer reports success, you need a layer that measures reality, not intent.
8. Backups are the difference between a story and an ending. Velero, covering objects and volumes, with a restore procedure someone had actually thought through. The weekend was long; without the backups it does not bear thinking about. Restore drills are cheap insurance against the worst day of your quarter.
9. Dangerous defaults outlive their context. The job's chart shipped with
DELETE_ARGOCD=true and WORKSPACES=* baked in — the configuration the incident ran
with. Every default on a destructive tool should be the safe one, with the dangerous
setting requiring an explicit, deliberate override.
Takeaways
The operator has been running the workspaces since — self-healing the envelopes, propagating the template, doing the two GitOps behaviors we actually needed. It has never deleted anything it should not have, partly because its design refuses read-modify-write races, and partly because we now know exactly what our destructive paths can cascade into.
The migration is the part of the story I would least like to repeat and the part I am most glad is written down. If you are planning a migration between two management systems, with thousands of live objects and a deletion step in the middle: test the clock, not just the code. Patch, don't rewrite. Abort on guard failure. Count what exists before and after. And keep your backups boring, complete, and rehearsed.
The alternative is a very quiet log stream, a green summary line, and a Monday morning full of empty namespaces.