← All posts

How one connection kills a database

An open transaction after an unhandled exception blocks a migration and every queued query—the GitHub and Postgres pattern.

How one connection kills a database
Contents

In brief

At GitHub, a database outage once came down to a single unclosed transaction: a schema change waited for an exclusive lock, new queries queued behind it, and everything stalled until someone killed the top connection. The same pile-up is easy to reproduce in Postgres—an unhandled app exception leaves a transaction open, a migration blocks, and a hot table stops responding.

What happened

Classic MySQL pattern: a schema change needs an exclusive table lock, but another session already touched the table and never COMMITted. The migration waits. Every new query on that table waits behind it. No deadlock, no automatic timeout—the chain sits until the bad connection is terminated.

In Postgres, a normal application user is enough. A transaction starts a SELECT on hot table orders, the app throws—no ROLLBACK. The session stays “in transaction” even though the query finished. A parallel ALTER TABLE waits for ACCESS EXCLUSIVE. Later SELECTs queue up. For a service expecting fast reads from orders, that’s downtime.

Often the trigger is default config: idle_in_transaction_session_timeout is off, so idle-in-transaction sessions never die on their own.

PlanetScale documents dashboard and CLI tools (pscale branch connections) to list and kill connections—even when connection slots are exhausted and you can’t run a debug query.

Why it matters

This isn’t a rare MySQL-only edge case. Any DB with table/schema locks can fall to one bad session. Symptoms look like total failure; the root cause is one connection that never ended its transaction.

GitHub used this scenario in DBA interviews. In production you meet it via pager duty on a Friday.

In practice

  1. Set idle_in_transaction_session_timeout in Postgres so idle transactions time out.
  2. On app exceptions inside a transaction: explicit ROLLBACK or pool cleanup that closes the connection.
  3. Plan hot-table migrations around locking; online ALTER on Vitess/PlanetScale reduces long exclusive locks on MySQL.
  4. Know how to inspect sessions and blockers before the incident, not only when slots are gone.
  5. When debugging, distinguish cancel query vs terminate transaction vs terminate connection—a finished SELECT with an open transaction often needs the last two.

Takeaway

One unhandled exception can take down a database. Idle-transaction timeouts, ROLLBACK discipline, and connection visibility are cheaper than an hour of hot-table outage.