What happened

Totum is a self-hosted low-code platform for internal web apps: records, cabinets, application tables, and custom logic, plus outbound HTTP calls. In the article this is not a lab install. It is a live system with PostgreSQL, user files, and integrations. The constraints were strict and ordinary: one main machine dying must not stop the system, the database must fail over by itself, the app must know where it is allowed to write, files must still be there after the switch, nobody edits DNS by hand, and the whole thing is rolled out with Ansible.

The witness carries almost no application load. It exists so etcd has quorum. PostgreSQL is run through Patroni: leader election and asynchronous replication. The application role is tied to that leader. A check asks whether the current node is Patroni’s primary; if yes, an “active” marker is created, if no, the marker is removed. The idea is short: whoever holds the database for writes is the only live application.

A layer-7 balancer knows both Totum nodes, but it asks GET /ha-health for health. The active node answers HTTP 200, the passive one answers HTTP 503. When Patroni moves the leader, the role check updates the marker, the balancer sees the new picture, and people land on the other node. DNS names are not rewritten.

Files do not ride along with Patroni. They are synced with lsyncd and rsync, and only in one direction: active to passive. After failover the direction flips with the role. The author calls two-way copying dangerous: both machines start writing to each other and you get conflicts. So only the passive node may accept changes. That is the fence.

A separate failure was not the database but the PHP-FPM pool. Slow outbound curl calls held workers, new requests waited, and the UI looked entirely dead. A temporary slow log with a 2-second threshold and a trace depth of 50 showed stalls on external calls, long polling, and notification checks. Timing around curl split the wait: connect was about 0.01 seconds, time to first byte was 76.4 seconds and 15.6 seconds. They were not waiting on the network to a neighbor. They were waiting on the other side’s answer. The pool was raised: up to 40 children, 20 at start and as spare, spare ceiling 40. A single slow request stayed slow, and it stopped taking the whole Totum down with it.

Why it matters

The textbook picture of high availability is three big servers, a cluster, and storage “as drawn”. Here the third machine stayed small, and the author names the scheme as a compromise. For an internal system you need to move off a single point of failure without inflating the bill, that is more useful than a diagram you cannot afford to run. The boundary is clear: there is quorum, there is one application writer, and files do not argue with each other.

The second lesson is wider than Totum. A database failover does not heal a pool that has all gone out to someone else’s HTTP. Once the workers are exhausted, the user does not care that PostgreSQL is up. Time to first byte separated “we connect slowly” from “the neighbor thinks for a long time”, and a larger pool contained the damage without making the foreign API faster. Those are different fixes, and they are easy to mix up if you only hear “everything is down”.

In practice

The article is one install, not a universal blueprint. Asynchronous replication means a hard crash of the leader can lose the latest writes; the author chooses automatic failover rather than synchronous writes to both nodes. Before copying the pool numbers, take your own slow log. Forty processes helped them because the bottleneck was a foreign answer, not the CPU.

The file fence is easy to break with a “convenient” two-way rsync. If both nodes believe they are active for a minute, the copies diverge. The active marker needs one source — for them, the Patroni leader — and the health check must look at that marker, not at the fact that a web process is merely running.

1. Budget a witness for etcd quorum if you are not ready to buy a third full server.
2. Tie the right to write in the app to the PostgreSQL leader, not to the fact that a web process is alive.
3. Give the balancer HTTP 200 only from the active node and HTTP 503 from the passive one, with no manual DNS.
4. Sync files one way, and reverse the direction together with the role.
5. If “everything is slow”, capture long-request traces and time to first byte on the outbound call before you grow the pool.

Takeaway

“Two servers and a witness” takes Totum off a single machine without turning the third node into another copy of the app. There is one live writer, files follow it, and people arrive where the health check is green. Beside that sat a separate loop: foreign slow answers exhausted the pool, and that was fixed by measurement and spare processes, not by another database server.