The Cost of the Clean Slate

The rain in Chennai does not fall; it hammers. At 3:14 AM, the sound of water hitting the AC compressor outside my window was interrupted by the high-pitched hum of my phone vibrating against the wooden nightstand.

It was Sundar. He was the on-call engineer, and he sounded tired, his voice thin over the cellular connection.

"Karthik, the primary is at 94.2% disk on the WAL volume. I had to drop the slot."

I sat up, the sleep clearing from my head. I did not blame him. Sundar’s decision was defensible, even logical under the circumstances. If db-primary-01 hit 100% disk utilization, PostgreSQL would PANIC and shut down, halting every transaction for our payment gateway. The culprit was the physical replication slot replicachennai01, which had stalled. The replica had stopped consuming WAL segments, causing the primary to pile up write-ahead logs on its local NVMe drive.

Sundar had executed the nuclear option:

SELECT pg_drop_replication_slot('replica_chennai_01');

It freed the primary from its obligation to preserve history for the dead replica. But dropping a physical replication slot does not trigger an instantaneous purge of WAL. Instead, it removes the retention pin. The segments are only marked as eligible for recycling. When the next database checkpoint occurred three minutes later, 120 gigabytes of WAL vanished from the primary's disk. The primary survived. The immediate crisis was averted.

But the replica was now an orphan.

My chest tightened, not because of Sundar’s action, but because of my own omission. Six months ago, I had written the Prometheus alert that woke him up. I had set the threshold, but I had neglected to write the runbook. The link in the alert payload pointed to a blank internal wiki page with a single line: TODO: Add recovery steps for replication lag. Sundar had acted without a map because I had failed to draw one.

"Is the WAL archiver running?" I asked, turning on my desk lamp.

"It says it is," Sundar said. "But the archive status directory has a backlog. I think S3 is rate-limiting us."

I checked my terminal. We ran a quick diagnostic. If the archived WALs were intact in our S3 bucket, we could have pointed the replica’s restore_command to the archive, allowing it to replay the missing history and catch up. But a check of the S3 bucket confirmed our fears: the archive was missing a continuous run of sixty-four segments between LSN 0/A3F1B000 and 0/B4000000 due to a credential rotation failure that had gone unnoticed during the previous afternoon.

There was no clever workaround here. There was no way to "bump" the timeline ID in pg_control to trick the replica into thinking it was synchronized. To do so would invite silent data corruption—a sin far worse than downtime. The replica’s data directory was functionally dead.

"We have to rebuild from scratch," I said.

"From the primary? Now?" Sundar asked. "During the monsoon traffic?"

"It is the only clean way," I replied. "We do it by the book."

We did not use custom scripts or unproven tools. We used pg_basebackup. Because the replication slot had been dropped, we had to ensure that we created a new slot during the backup process itself. If we ran a standard backup and created the slot afterward, the primary might recycle the WALs generated during the backup before the replica could start streaming them.

I walked Sundar through the command over SSH:

pg_basebackup -h db-primary-01.internal -D /var/lib/postgresql/15/main \
  -U replication_user -P -v -X stream -C -S replica_chennai_01

The -C and -S flags were the critical pieces. They instructed pgbasebackup to create a new physical replication slot named replicachennai_01 on the primary before beginning the data transfer, securing our LSN starting point.

We sat in silence as the network interface saturated. On my screen, the progress bar crept forward.

34215/948120 MB (3%)

There are no heroics in database reliability engineering. There are only cold configurations, completed runbooks, and the slow transfer of bytes across a wire. I poured myself a cup of yesterday's tea, cold and bitter, and watched the numbers change. Outside, the rain continued to beat against the glass.