Kappsul is an educational platform built at the University of Lorraine. A student picks a service from a catalog, gets a Kubernetes namespace, and a dashboard shows what is running inside it. That dashboard is the product: it has to be live. It also has to survive being open in three thousand tabs at the same time.
This is the story of the small distributed system we put between the browser and the Kubernetes API so that those two statements could both be true.

The trap: a session is a terrible unit of scale
"Live" sounds like a UI concern. In a Kubernetes-backed product it is an API server concern, and the obvious implementations fail in two very different ways.
Polling. Each dashboard asks the backend for the current state every few seconds. With three thousand open dashboards and a five-second poll, that is roughly six hundred requests per second before anyone clicks anything. Every response is a full read: list the deployments, the statefulsets, the pods, the resource quota. The payload grows with the number of objects, and the data is stale by definition. You have built a load generator against your own control plane, and you still get to show users numbers that are a few seconds wrong.
One watch per browser. Kubernetes gives you a streaming API, so the reflex is to open one stream per tab. A single dashboard needs more than one stream: deployments and statefulsets for a workspace, or pods for a specific app, or namespaces for the admin view. Three thousand users becomes six to nine thousand long-lived streams, each one a connection the API server has to hold, buffer, and fan out to. The API server is a shared, scarce resource: every stream it serves for your UI is capacity it cannot spend running the cluster.
There is a third, subtler trap: running one watch per backend pod without coordination. You scale the service to ten replicas, and now every replica opens its own watch for the same namespace. The UI looks fine. The API server sees ten identical streams. Scaling your own service multiplies load on the component you were trying to protect.
The mistake in all three cases is the same: we made the watch a property of the user. A watch is a property of the data. If a thousand people are looking at the same workspace, there is no reason for the API server to know about more than one of them.
Watch 101, in one page
A Kubernetes watch is a streaming delta feed over a collection. You first list the
collection to get a metadata.resourceVersion, then you watch from that version. The
server sends ADDED, MODIFIED, DELETED and ERROR events; every object carries the
resourceVersion you need to resume. The watch is bounded by a timeout, and when the
server closes it — on timeout, on error, or because the underlying history expired — the
client is expected to rewind and watch again from the last version it saw.
Two details do most of the work in a design like ours:
- Selectors. A watch is only as expensive as the set it observes. Namespace scoping
and label selectors (
app.kubernetes.io/instance=my-app,meta.kappsul.io/user-ns=yes) cut the stream down to exactly the slice the UI needs. - Bounded duration. A watch that runs forever is a watch nobody re-evaluates. A bounded watch is a natural checkpoint: when the window closes, you can ask whether anyone still cares before opening the next one.
If you have written a controller, you know the standard answer: an informer. One watch per resource type per process, an in-memory cache, and handlers fanning out to whoever needs the data. Informers are the right building block for a controller. They are not a complete answer for a multi-tenant web app, because an informer is process-local. Ten replicas still mean ten informers, and you do not want one informer per end user either. The thing you actually need to coordinate is the watch, and the unit of coordination is not the connection or the process. It is the scope.
The unit of sharing: a scope
A scope is the slice of the cluster that a group of viewers has in common. In Kappsul we have three:
| Scope | Kubernetes objects | Selector | Who looks at it |
|---|---|---|---|
allWorkspaces | Namespaces | meta.kappsul.io/user-ns=yes | Admin views |
workspace:{name} | Deployments, StatefulSets | namespace-scoped | Everyone on one workspace dashboard |
app:{workspace}:{app} | Pods | app.kubernetes.io/instance={app} | Everyone on one app detail page |
That is the whole trick, expressed as a table. A scope is a promise: these viewers are looking at the same data, so they can share the same stream. The number of watches the API server sees becomes the number of active scopes, not the number of users.
Back-of-the-envelope: if each dashboard needed three streams and every user got their own, three thousand users would mean nine thousand watches. If those users are spread over six hundred workspaces, and the admin view collapses to a single global scope, the system needs roughly six hundred and one. The users are a dimension of the UI, not of the watch.
Notice what is not in the diagram: an arrow from each browser to the API server. The only thing that touches Kubernetes is the single owner of the scope.
One owner per scope
The coordination layer is a small Go package, watch-manager, shared by the admin,
workspace, and application services. It does not know anything about Kubernetes. It
implements a lease: exactly one container instance owns a scope at a time, and the lease
is stored in Redis under a key derived from the scope.
watch-all-workspaces
watch-workspace:{workspace}
watch-app:{workspace}:{app}
Starting a watch is a three-way decision: join, claim, or take over.
The join path is the one that makes thousands of users cheap. A second viewer does not create a second watch; it proves that the scope is still wanted by pushing the lease forward, and returns the existing owner.
// watch-manager/manager.go (trimmed)
watchKey := handler.GenerateWatchKey(watchCtx)
currentWatcher, _ := c.redisClient.Get(ctx, watchKey).Result()
if currentWatcher != "" {
alive, _ := c.isContainerStillWatching(ctx, currentWatcher, config)
if alive {
// Someone already watches this scope: join it, do not duplicate the stream.
ttl := time.Duration(config.WatchRedisTTL) * time.Second
c.redisClient.Set(ctx, watchKey, currentWatcher, ttl)
return &WatchResult{Success: true, WatchingContainerID: currentWatcher}, nil
}
// The owner is gone: exactly one contender wins the takeover.
taken, _ := c.attemptTakeover(ctx, currentWatcher, watchKey, config)
if !taken {
return &WatchResult{Success: false, Message: "takeover lost"}, nil
}
}
// Nobody owns the scope: claim it, then start the actual watch.
ttl := time.Duration(config.WatchRedisTTL) * time.Second
c.redisClient.Set(ctx, watchKey, c.containerID, ttl)
c.startHeartbeat(watchKey, config)
if err := handler.StartWatch(ctx, watchCtx); err != nil {
c.stopHeartbeat(watchKey)
c.redisClient.Del(ctx, watchKey)
return nil, err
}
Proving the owner is alive
Ownership alone is not enough: a pod can die while still holding the key. The owner proves liveness with a heartbeat, and every contender checks it before trusting the owner. The heartbeat is written every thirty seconds with a sixty-second TTL, and a heartbeat older than sixty seconds is treated as dead.
// A per-container heartbeat, refreshed by the owner's active watch loops.
heartbeatKey := config.HeartbeatPrefix + c.containerID
ticker := time.NewTicker(time.Duration(config.HeartbeatInterval) * time.Second)
go func() {
for range ticker.C {
ts := strconv.FormatInt(time.Now().Unix(), 10)
c.redisClient.Set(ctx, heartbeatKey, ts,
2*time.Duration(config.HeartbeatInterval)*time.Second)
}
}()
// ...and on the reader side:
timeSinceHeartbeat := time.Now().Unix() - timestamp
return timeSinceHeartbeat < 60, nil
Taking over without a stampede
When the owner is declared dead, every replica that notices may try to take over. The claim has to be atomic, or you get the thing you were trying to avoid: several pods each convinced they own the scope, each opening a watch. We do the compare-and-swap in Redis with a Lua script, so the check and the write happen as one operation.
-- takeover.lua: only replace the owner we all agreed was dead.
local current = redis.call('get', KEYS[1])
if current == ARGV[1] then
redis.call('set', KEYS[1], ARGV[2], 'EX', ARGV[3])
return 1
else
return 0
end
One contending container gets 1, starts the watch, and begins a heartbeat. The others
get 0 and back off. The winner also deletes the dead owner's heartbeat key, so the next
contender sees a clean state.
The lease is the source of truth, not the watch
The most useful shift in the whole design was to stop thinking of the watch as a resource that is started and stopped, and to think of it as a lease that is kept alive.
Stopping a watch does not delete the key. It lowers the TTL to sixty seconds and stops the heartbeat. That grace period matters for three reasons:
- another user may still be viewing the same scope;
- the same user may be reloading the page or reconnecting through a flaky network;
- killing the stream immediately would guarantee a cold start on the next request.
The frontend is the other half of the lease. A React hook starts the watch on mount, renews it every two minutes while the page is open, and stops it on unmount. The renewal is not a new watch; it is the same "join" path the backend already supports, which extends the lease if the owner is alive and takes over if it is not.
// useWatch.ts (trimmed)
const { enabled = true, renewalInterval = 2 * 60 * 1000 } = options;
useEffect(() => {
if (!enabled) return;
startWatch(); // claim or join the lease
const timer = setInterval(renewWatch, renewalInterval); // keep the lease fresh
return () => {
clearInterval(timer);
stopWatch(); // drop the lease, not the stream
};
}, [enabled, config.type]);
That gives the system three independent clocks, and it is worth being explicit about what each one protects:
| Clock | Value | Where it lives | What it protects |
|---|---|---|---|
| Kubernetes watch window | 90s | stream to the API server | bounds each stream and forces a rewind |
| Heartbeat interval | 30s | owner pod to Redis | proves the owner is alive |
| Heartbeat staleness | 60s | Redis / reader | declares the owner dead |
| Lease TTL | 180s | Redis | keeps ownership while the scope is viewed |
| Client renewal | 120s | browser | keeps the lease fresh while the page is open |
| Graceful stop | 60s | Redis | lets reconnects and other viewers win |
The ordering is not accidental. The client renews before the lease expires. The heartbeat is much more frequent than the lease, so a dead owner is noticed long before the lease itself runs out. The Kubernetes watch window is independent of both: it is there to make the stream finite, not to keep ownership.
The watch itself: bounded, rewound, and deliberately boring
The watch-manager decides who watches. The service decides what to watch, and it
does so with a manual rewatch loop. For a workspace, that means deployments and
statefulsets in one namespace, each rewound from the last resourceVersion it saw.
// workspace-service, WatchWorkspaceResources (trimmed)
for {
deploymentWatcher, _ := ws.k8sClient.WatchDeployments(ctx, workspace, metav1.ListOptions{
ResourceVersion: deploymentResourceVersion,
TimeoutSeconds: &timeout, // 90 seconds
})
statefulSetWatcher, _ := ws.k8sClient.WatchStatefulSets(ctx, workspace, metav1.ListOptions{
ResourceVersion: statefulSetResourceVersion,
TimeoutSeconds: &timeout,
})
watchLoop:
for {
select {
case event, ok := <-deploymentWatcher.ResultChan():
if !ok {
statefulSetWatcher.Stop()
break watchLoop // window closed: rewind with the last version
}
if d, ok := event.Object.(*appsv1.Deployment); ok {
deploymentResourceVersion = d.ResourceVersion
}
ws.emitWorkspaceChanged(ctx, workspace, "watch-deployment")
case event, ok := <-statefulSetWatcher.ResultChan():
// same shape for StatefulSets
case <-ctx.Done():
deploymentWatcher.Stop()
statefulSetWatcher.Stop()
return
}
}
time.Sleep(time.Second) // small backoff before rewinding
}
Two things are worth calling out.
The rewind is where you handle the ugly parts. A watch can close for an ordinary
reason (timeout) or a bad one (ERROR, expired history). Carrying the last
resourceVersion across the rewind avoids gaps in the common case. If the API server
answers that the version is too old — the 410 Gone case — the correct recovery is not
to retry the same version forever but to list again and start fresh. The bounded window
exists partly to make that decision cheap: you are never more than ninety seconds away
from a clean restart.
The selectors are part of the contract, not an optimization. The admin watcher lists
namespaces with meta.kappsul.io/user-ns=yes. The app watcher lists pods with
app.kubernetes.io/instance={app} and watches from the list's resourceVersion. The
workspace watcher stays inside one namespace. If you find yourself watching a broad
collection and filtering in memory, you have moved the cost from the API server to your
own process and kept the worst part of the stream.
Fan-out: send signals, not objects
At this point the Kubernetes side is settled: one stream per active scope. The remaining question is how one stream becomes updates for three thousand browsers.
The short answer is that the browser never sees a Kubernetes object. It sees a small signal, and then it reads the truth through the normal API.
The service emits a topic event with a routing key in the client.* namespace. The
gateway owns a durable client queue bound to client.# on the events exchange. It
consumes the event, turns the routing key into a GraphQL subscription name, and publishes
to Redis Pub/Sub. Redis is what makes the gateway horizontally scalable: a WebSocket may
be connected to any replica, and the event still reaches it.
// gateway: routing key -> subscription name
private getSubscriptionName(routingKey: string): string {
return routingKey
.split(/[._-]/)
.filter((part) => part !== 'client')
.map((part, i) => (i === 0 ? part : part[0].toUpperCase() + part.slice(1)))
.join('');
}
// client.workspace-resources.changed -> workspaceResourcesChanged
The payload is intentionally tiny: a workspace name, an app name, and a phase such as
initial, watch-deployment, or watch-statefulset. It is a cache invalidation
message, not a data transfer. The frontend subscribes, filters by the arguments
(workspace, and for apps also app), and refreshes through the same query it would
have used on first load.
// The event is a signal. Read the truth through the API.
useEffect(() => {
if (subscriptionData?.workspaceResourcesChanged) {
refetch();
}
}, [subscriptionData, refetch]);
This choice buys three things:
- Bandwidth and coupling. Kubernetes objects are large and versioned; a change signal is neither. The browser does not need to understand Deployments to know that the deployments changed.
- Authorization stays on the read path. The event says "something changed"; the refetch goes through the normal authenticated GraphQL query, with the normal access rules.
- Consistency stays where it already lives. The query path knows how to assemble the dashboard from the database and the cluster. The event path does not have to reimplement that assembly.
The cost is refetch amplification: one real change can trigger one refetch per connected viewer. That is usually the right trade at this scale — a refetch is a cheap, cacheable read, while a stream per user is not — but it is a cost you should watch, not ignore.
Failure is the normal case
The design assumes pods die. When the owner of a scope dies, nothing is deleted; the lease simply stops being refreshed.
A heartbeat older than sixty seconds is enough for another replica to attempt a takeover, and the Lua compare-and-swap makes sure only one succeeds. The practical failure window is on the order of a minute, plus however long it takes for a renewal or a startup scan to notice.
Startup recovery uses the same primitives. Each service scans its own prefix in Redis, restores the scopes it already owns, and attempts to take over scopes whose owner is dead. That is why a rolling restart does not produce a thundering herd of cold starts: the new pod does not blindly claim every scope it sees; it checks the heartbeat first.
The system is deliberately eventually consistent during a failover. For a short window, two watchers can observe the same scope. That is acceptable because the events are idempotent invalidation signals: the UI refetches and converges. A brief duplicate watch is a much smaller problem than a coordination protocol that blocks on a dead process.
What this design costs
It would be dishonest to present this as free. The API server load did not disappear; it moved, and the new places are cheaper and easier to scale independently, but they are still places.
- Redis is in the hot path. Every start, renewal and liveness check is a Redis round-trip. It is a small number of operations per scope, not per user, but Redis is now part of your availability story. Treat it as such: Sentinel or a managed deployment, and a plan for what happens when it is slow.
- Liveness depends on clocks. The heartbeat check compares timestamps written by different pods. Reasonable clock discipline is enough, but a pod with a badly skewed clock can look dead or immortal. A purely TTL-based liveness signal is a possible hardening step.
- Refetches can stampede. A burst that touches five hundred pods produces a burst of invalidations, and then a burst of refetches. Debouncing or coalescing per scope, and jittering client refreshes, are the obvious next levers.
- Duplicate watches exist briefly. Failover is not atomic across the API server. Accept it, and make sure duplicate streams cannot cause duplicate side effects.
- There is more surface to operate. Redis keys, heartbeats, lease TTLs, a message broker, a pub/sub layer and WebSocket replicas. Each one is simple; the combination is a system you have to observe.
The trade is still strongly in favor of this design, because the resource it protects — the Kubernetes API server — is the one shared by everything else in the cluster.
Tuning the knobs
The numbers are not magic. They encode a few simple relationships:
| Knob | Kappsul value | Constraint |
|---|---|---|
WatchExpirationTime | 90s | the window between rewinds |
WatchRedisTTL | 180s | must exceed the client renewal interval |
HeartbeatInterval | 30s | must be well under the staleness threshold |
| Heartbeat staleness | 60s | must be under the lease TTL |
| Client renewal | 120s | must be under WatchRedisTTL |
| Graceful stop TTL | 60s | short enough to free the scope, long enough to survive a reload |
If you take one rule from this table, take this one: the client must renew before the lease expires, and the heartbeat must expire before the lease does. Get that ordering wrong and you will either drop live watches or keep dead ones.
What I would measure next
The design is only defensible with numbers, and the numbers that matter are mostly on the API server side:
- Watches per resource type, and how that number tracks active scopes rather than connected users. This is the core claim of the design.
- List and watch QPS, plus the rate of
410 Goneresponses. A rising 410 rate means your rewind is too slow or your windows are too long. - Event delivery latency, p50 and p99, from the Kubernetes event to the browser refetch. This is what users actually feel.
- Refetch rate per scope. This is where a burst turns into a stampede.
- Failover time, from owner death to a new owner watching. The heartbeat and lease values above imply a target; measure whether you hit it.
If I were hardening this further, I would start with per-scope debouncing of invalidation signals, jitter on client renewals, and a stricter liveness check that does not depend on comparing wall clocks across pods.
Takeaways
- Make the watch a property of the data, not the user. Define scopes — resource set plus selector plus namespace — and share one stream per scope.
- Elect one owner per scope. A Redis key with a TTL is enough, provided you add a heartbeat so a dead owner does not hold it forever.
- Make takeover atomic. A Lua compare-and-swap prevents the failover stampede that would otherwise recreate the problem you solved.
- Treat the watch as a lease. Start on view, renew while viewed, lower the TTL on stop, and let it expire naturally.
- Bound and rewind the stream. A finite window forces you to re-evaluate demand and
gives you a clean place to handle expired
resourceVersions. - Fan out signals, not objects. Let one Kubernetes event become one small invalidation, and let each client read the truth through the normal API.
- Assume failure. Recover scopes on startup, accept brief overlap, and make duplicate events harmless.
- Protect the API server first. It is the shared resource; everything else in this design exists to keep it out of the hot path.