Skip to content
pistra

Spread a cluster across zones

Three voters in three zones survive the loss of a zone. Three voters in one zone do not, and nothing in the install says which one you got. The chart asks for a zone spread by default, as a preference rather than a requirement, so this guide is about checking that the preference was honoured and deciding what to do when it cannot be.

What a cluster survives has the reasoning, including why a cross-zone commit does not land on the request path.

Before you start

Section titled “Before you start”

The spread reads topology.kubernetes.io/zone off the nodes. Managed clusters set it. A cluster built by hand may not.

Run this command to see the label and the zones you actually have:

Terminal window
$ kubectl get nodes -L topology.kubernetes.io/zone

An empty column means every node is one zone as far as the scheduler is concerned, and the constraint cannot do anything until the label is there.

1. Check where the voters landed

Section titled “1. Check where the voters landed”

The constraint is whenUnsatisfiable: ScheduleAnyway, so a cluster that cannot honour it schedules the pods anyway and reports nothing. Read the placement rather than assuming it.

Run these two commands and compare their output:

Terminal window
$ kubectl get pods -n pistra-system -l app.kubernetes.io/name=pistra \
    -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName
$ kubectl get nodes -L topology.kubernetes.io/zone

Three pods on nodes in three different zones is the shape that survives one. Any other answer is a cluster that loses its quorum when one zone goes, and it will do so without ever having warned you.

2. Decide whether the spread is a requirement

Section titled “2. Decide whether the spread is a requirement”

ScheduleAnyway exists so a single-node development cluster comes up instead of hanging with pods pending. In production the same setting lets a scheduling shortage quietly become a quorum on one failure domain.

Set the zone constraint to DoNotSchedule when you would rather have a pod pending than a quorum in one zone:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels: {}
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels: {}

An empty matchLabels is filled in with the release’s own selector, so the constraint spreads this release’s pods and not everything in the namespace.

That choice has a cost, and it is the one worth stating plainly. A zone that is short of capacity now leaves a voter Pending rather than placing it somewhere available, and a two-voter cluster tolerates no failures at all. DoNotSchedule is right when you have capacity in three zones and want to be told when you stop having it. It is wrong when you would rather be up.

3. Rehearse the zone loss

Section titled “3. Rehearse the zone loss”

The placement is a claim about a failure you have not had yet. Cordon and drain the nodes of one zone to have it.

Run this against each node in a single zone, one at a time:

Terminal window
$ kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

With one voter per zone, pdb.maxUnavailable: 1 permits that eviction and the drain completes. pistra_cluster_leader_known stays 1 on the surviving pods, and minting a key still works.

A drain that stalls on the PodDisruptionBudget is the budget telling you two voters share the zone you are draining. That is the answer to step 1, arrived at the hard way. Uncordon, fix the placement, and try again.

A rehearsal on a production cluster is a real disruption. Run it where an outage is affordable, or during a window where one is.

Two zones is not three

Section titled “Two zones is not three”

A cluster with two zones and three voters puts two voters in one of them. Losing that zone loses the quorum, so the deployment survives one of its two zones and not the other. The arithmetic does not care which one you call primary.

Three voters need three zones. A region with two is a region where the honest options are accepting the asymmetry, or running the deployment somewhere with three.

Section titled “Related”