Field Guide

Replica Scheduling Failure in Longhorn

Your volumes are stuck Degraded because the bundled StorageClass overrides your replica count — here's the two-minute diagnosis and the durable fix.

Field guide~6 min readIntermediate

The short version: You set Longhorn's global default-replica-count to 2 because you have two nodes. Your volumes still come up Degraded, because the bundled longhorn StorageClass hardcodes numberOfReplicas: "3" — and StorageClass parameters outrank the global setting. Don't edit the bundled class (Longhorn regenerates it). Author your own class and name it explicitly. Tested on Longhorn v1.13.0, two-node k3s; the "3" was verified in that release's official deploy manifest, and the precedence mechanism is standard Longhorn behavior.

The trap

The symptom

A fresh volume attaches and works. Reads and writes are fine. But the Longhorn UI shows the volume Degraded, with Replica Scheduling Failure — two of three replicas Running and in sync (RW), the third stopped and unschedulable. And it never heals: not in two minutes, not in two days. Nothing is broken. The volume is faithfully trying to satisfy a spec that asks for three replicas on two nodes, and replicas can't double up on a node.

That's what makes this trap expensive. It looks like a transient warm-up state, so you wait — then you debug networking, disk pressure, and node health. Everything except the replica count stamped on the volume at creation time.

Check what your volume actually asked for:

kubectl -n longhorn-system get volumes.longhorn.io
# ROBUSTNESS reads "degraded" for the affected volume

kubectl get storageclass longhorn -o jsonpath='{.parameters.numberOfReplicas}'
# prints: 3 — even though your global default says 2

Degraded + Replica Scheduling Failure with healthy, in-sync replicas = suspect the count, not the cluster.
The volume's spec is the ground truth — read it before debugging anything else.

The cause

Why it happens: three layers

The replica count for a new volume is decided by the first of these layers that speaks:

The volume's own spec — set at creation, wins over everything.
The StorageClass parameter numberOfReplicas.
The global setting default-replica-count.

Most install guides set layer 3 and stop. But Longhorn's own deploy manifest ships a longhorn-storageclass template with layer 2 already filled in — numberOfReplicas: "3" — and that parameter silently overrides your global 2 for every volume created through the bundled class. On a cluster with three or more nodes you never notice. On two nodes (or one), every volume is born degraded.

Settings in layered systems don't merge — they override, in a specific order.
The layer you didn't look at is the one that wins.

The fix — new volumes

The fix for new volumes: your own StorageClass

Footgun: Do not edit the bundled longhorn class. Longhorn's driver-deployer regenerates it from its own template, so your edit quietly reverts and the trap re-arms itself.

Author your own class — identical to the bundled template except the name and the count — and reference it explicitly in your PVCs:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn-rep2
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: "Delete"
volumeBindingMode: Immediate
parameters:
  numberOfReplicas: "2"
  staleReplicaTimeout: "30"
  fromBackup: ""
  fsType: "ext4"
  dataLocality: "disabled"
  unmapMarkSnapChainRemoved: "ignored"
  disableRevisionCounter: "true"
  dataEngine: "v1"
  backupTargetName: "default"

Two deliberate choices in that file. First, no default-class annotation — this class is opt-in, so a workload gets replicated storage by naming storageClassName: longhorn-rep2, not by accident. (A Longhorn install also claims the cluster default-class flag for the bundled class, so PVCs that name no class at all still land on the 3-replica template until you change that too.) Second, everything else is copied verbatim. The point is to change exactly one variable — a hand-rolled class with "improved" parameters is a new experiment, not a fix.

Own class, explicit name in every PVC, one variable changed.
Never patch the bundled class — it regenerates.

The fix — existing volumes

Fix the volume you already have

A StorageClass only stamps volumes at creation. Changing classes does nothing for a volume that already exists with 3 in its spec — it keeps asking for a third replica. Fix the volume itself:

kubectl -n longhorn-system patch volumes.longhorn.io <volume-name> \
  --type merge -p '{"spec":{"numberOfReplicas":2}}'

Or in the Longhorn UI: volume → ⋮ menu → Update Replica Count → 2. Within moments the volume's robustness flips to healthy, and Longhorn sweeps the unschedulable third-replica object away on its own. In the build this guide came from, the two real replicas were RW and in sync the whole time — no data was ever at risk. Only the count was wrong.

Footgun (Windows operators): Run that patch from bash, not PowerShell 5.1. PowerShell strips inner double quotes from native-command arguments that contain no spaces, so the JSON arrives mangled (invalid character 's' looking for beginning of object key string). It failed twice that way in the original build, then worked first try from bash.

Existing volume? Patch its spec (or use Update Replica Count) — the class change alone won't heal it.
JSON patches go through bash. PowerShell 5.1 will eat the quotes.

The proof

Prove it before you trust it

A healthy-looking volume is a claim; a surviving file is evidence. Before a real workload lands on the new class, run three steps in a throwaway namespace:

Create a PVC on your new StorageClass, mount it in a scratch pod, and write a timestamped file.
Delete the pod and recreate it. The file must come back untouched — compute is disposable, data persists.
Cordon the node the volume is attached to, recreate the pod again, and confirm it lands on the other node with the file — and its original timestamp — intact.

In the original build, the proof file's timestamp came back character-identical after a pod kill and a forced cross-node move. That's when the storage layer stopped being "installed" and started being trusted.

Write, kill, cordon, re-read. Four verbs, one verdict.
Don't skip the cordon step — a move test with one healthy replica proves nothing.

The lesson

The lesson worth keeping

When a system insists on a number you never set, don't debug the machinery — find which layer stamped it. That holds well beyond Longhorn: defaults files under config files under flags under API specs, in that order, in almost every tool you run.

The dangerous checklist

Read a degraded volume's spec before touching the cluster.
Check any bundled StorageClass's parameters against your global settings.
Ship your own class with exactly one variable changed.
Patch existing volumes' specs — classes only stamp new ones.
Prove replication with a write, a pod kill, and a cordon before trusting it.

From a live install record, Longhorn v1.13.0 on a two-node k3s cluster. Node names, volume IDs, and addresses from the original build are omitted; every command and parameter above is reproduced exactly.