Recover a cluster that has lost quorum

Enough voters are gone for good that the survivors can never elect a leader. Reads still serve from each node’s local state, so traffic keeps flowing on whatever configuration was last applied, but nothing commits. No key can be minted, no budget deltas land, no configuration source can be written, and no provider can be suspended. Every door out of this is itself a write. That makes it different from every other failure in this documentation.

pistra cluster recover is the way out. It rewrites one stopped node’s raft state so that it believes in a smaller membership, and that node can then elect alone and start committing again. Keys, budget ledgers, certificates, suspensions and configuration sources on that disk are kept.

This is not a repair. It forces a configuration the old cluster never agreed to, and it commits log entries that may never have been committed. Reach for it when members are gone permanently, from a destroyed volume or a deleted node pool, and not when they are merely down. A member that is down comes back on its own. A member that has lost only its disk rejoins through cluster.join. Neither needs this.

Before you start

Recovery runs against a stopped node’s files. The data directory is locked while the gateway holds it, and the command says so rather than waiting:

pistra: cluster: data directory is locked by a running process: /var/lib/pistra/raft/raft.db

1. Look, on every survivor

Without -confirm the command writes nothing. Run it on each survivor first:

pistra cluster recover -config /etc/pistra/config.yaml

Use the following example:

node pistra-1, data directory /var/lib/pistra/raft

committed configuration: 3 members, 3 voting
  keep  pistra-1  pistra-1.internal:7000  voter  <- this node
  DROP  pistra-2  pistra-2.internal:7000  voter
  DROP  pistra-3  pistra-3.internal:7000  voter

log: last index 10422, term 7; newest snapshot at 8192
     Recovery commits all 2230 entries past the snapshot, including any
     the old cluster never committed: a write whose client already saw
     an error can take effect here.
     Compare this last index against the other survivors before choosing.
     Recovering the node that is behind discards what the node that is
     ahead still holds.

Dropping pistra-2 and pistra-3 asserts they are gone for good. ...

Nothing has been written. Re-run with -confirm to recover.

Decide two things from that output. They are why this is a command you read rather than a file you drop in place:

Which node. last index is the only thing that distinguishes two survivors. Recover the highest one. Recovering a node that is behind silently discards what the node ahead of it still holds, because that node comes back empty and is refilled from the one you chose.

Which members. The default keeps only this node. -keep pistra-1,pistra-2 keeps more, and the addresses come off this disk. Recovery can only drop members, never add one or change an address, so there is nothing to transcribe. Two survivors that can both be reached are worth keeping together. Recover with both, and no rejoin is needed. If you are not sure the second one will come back, keep one and rejoin it later.

2. Recover, once

Use the following example:

pistra cluster recover -config /etc/pistra/config.yaml -confirm

Then start the gateway. It elects immediately and the admin API begins accepting writes again.

If you kept more than one member, run the command on each of them with the same -keep list before starting any of them.

3. Erase the dropped nodes

A dropped member that starts again with its data directory intact does not rejoin. It still holds the old configuration. If enough of the dropped members come back together, they have the numbers to elect among themselves. That is a second cluster, serving traffic and sharing this one’s secret. Nothing on the recovered side can see that happen or prevent it.

So the drop is a declaration, and it has to be made true:

Kubernetes: kubectl delete pvc data-pistra-2
VMs: delete cluster.data_dir before the host runs the gateway again

One returning node on its own cannot reach the old quorum, so it spins on elections and never finishes starting. It does not disturb the recovered cluster. Pre-vote keeps it from inflating anyone else’s term, and a node outside the configuration is not voted for. That is a reprieve, not a safety property. Erase them.

4. Grow back

Replacements join the ordinary way, with an empty data directory:

cluster:
  node_id: pistra-2
  data_dir: /var/lib/pistra/raft
  join: ["pistra-1.internal:7001"]   # a survivor's forward address

They come in as non-voters, catch up, and are promoted, the same path every scale-up uses. Recovering to one node and rejoining the rest is usually better than recovering to several. On Kubernetes, scaling the StatefulSet back up does this for you once the old PVCs are gone.

On Kubernetes

The image has no shell, so recovery runs as a one-off pod that mounts the survivor’s volume. Scale to zero first. The volume is ReadWriteOnce and the running pod holds the lock either way.

kubectl scale statefulset/pistra --replicas=0
kubectl wait --for=delete pod/pistra-0 --timeout=120s

Add this configuration:

apiVersion: v1
kind: Pod
metadata:
  name: pistra-recover
spec:
  restartPolicy: Never
  containers:
    - name: recover
      image: <the image and tag this release is running>
      args:
        - cluster
        - recover
        - -data-dir=/var/lib/pistra/raft
        - -node-id=pistra-0
        # - -confirm
      volumeMounts:
        - name: data
          mountPath: /var/lib/pistra
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-pistra-0

kubectl logs pistra-recover is the plan. Read it, uncomment -confirm, re-apply, then:

kubectl delete pod pistra-recover
kubectl delete pvc data-pistra-1 data-pistra-2   # the dropped members
kubectl scale statefulset/pistra --replicas=3

-data-dir and -node-id are passed directly here rather than -config, because the recovery pod does not mount the ConfigMap. It does not need the ConfigMap, because the node id is its ordinal name and the data directory is fixed by the chart.

What recovery is not

Not a way to change addresses. It writes back what it read. A node whose address changed is a new node. Wipe it and rejoin.
Not a bootstrap. It refuses a data directory with no raft state in it, because an empty directory is nearly always the wrong directory. A new cluster is cluster.bootstrap.
Not reversible. The old configuration is gone once it is written, and the log is compacted into the new snapshot. That is what the dry run is for.
Not for a node that is merely behind. Followers catch up on their own, from a snapshot if they have to.