The Weight of the Wire

The rain in Vikhroli does not fall; it hammers. It beats against the double-paned glass of the tech park with a low, wet rumble that matches the vibration of the server rack in the corner of my home office. On the second monitor, the disk space alert for the primary database node in Mumbai was still glowing amber.

Karan’s voice came through the Slack call, thin and raspy. In the background, the rhythmic, static-heavy breathing of his six-month-old daughter came through his baby monitor.

"I had to make the call, Shamik," Karan said. He cleared his throat twice, a nervous habit he’d had since his intern days. "The primary was at ninety-four percent disk usage on a four-terabyte volume. If the write-ahead logs filled the mount, the engine would have crashed. The API gateway would have died before the morning shift started. I dropped the replication slot."

I rolled an unlit clove cigarette between my index finger and thumb, tasting the paper. "I know why you did it."

"The disk dropped to forty-two percent within three minutes," he said, his voice pleading for validation. "It freed over two terabytes of WAL. It worked."

"It worked for the primary," I said. My voice was flatter than I wanted it to be. "But dropping the slot removed the retention barrier. The moment the checkpoint ran, those two terabytes of WAL were marked as eligible for recycling. They’re gone, Karan. The Bangalore standby is dead-ended. It’s missing three hundred thousand segments. There’s no continuous chain left to replay."

There was a long silence on the line, save for the hum of the baby monitor. Karan knew the math as well as I did. You cannot patch a gap in physical replication. The standby cannot skip ahead; it has no mechanism for ignorance. To fix it, we had to stream the entire state of the primary database across the country over a commercial network while the primary was still taking live production traffic.

My chest tightened. The fault wasn’t entirely Karan’s. Eight months ago, I had written the Prometheus alert rule that woke him up tonight: PostgresReplicationLagHuge. But when I created the alert, the ticket for the corresponding runbook had been marked as low priority. I had left the runbook URL in the alert template pointing to a blank page in our internal wiki. I had gone to sleep that night feeling accomplished because the metrics pipeline worked. I had left the human being in the dark.

"We need to rebuild it," I said.

"Now? During the monsoon batch runs?"

"If we wait until 8:00 AM, the network will be saturated. We do it now."

I opened my terminal and SSH’ed into the remote standby server in Bangalore. The prompt hovered, green against black.

ssh admin@10.120.4.12
sudo systemctl stop postgresql
sudo -u postgres rm -rf /var/lib/postgresql/15/main/*

Deleting the data directory of a database—even a broken replica—always feels like a small death. The disk usage indicator for the standby plunged to zero.

Now came the physical tax of our shortcut. We had to run pg_basebackup. Because Karan had dropped the replication slot on the primary, we couldn’t just pull the raw bytes; we had to recreate the slot on the primary at the exact moment the backup started to ensure the transaction log didn’t drift during the transfer.

I typed out the command, staging the parameters with deliberate care. I couldn't afford a syntax error. If I missed the creation flags, the backup would finish and find itself immediately out of sync again.

sudo -u postgres pg_basebackup \
  -h 10.120.2.10 \
  -D /var/lib/postgresql/15/main \
  -U replicator \
  -P -v \
  -R \
  -C -S standby_mumbai_slot

"Look at the flags," I told Karan. "We use -C to create the slot, and -S to name it. The -R is what writes the standby.signal file and configures the connection back to the primary. If we don't do this in a single pass, we’re back here at sunrise."

I pressed Enter.

On the screen, the progress bar appeared.

3420/2184560 MB (0%)

The transfer speed hovered at forty-eight megabytes per second. Over a two-terabyte database, the transfer would take twelve hours if the connection remained stable. Twelve hours of high read IOPS on our primary database while Mumbai’s logistics fleet began their morning runs.

"The primary's CPU is climbing," Karan whispered. "It’s at forty-five percent. Shamik, if the API response times spike, the automated routing engine will fail."

"It will hold," I said, though my hand was damp against the desk.

We sat in the dark, separated by eight hundred kilometers of copper and fiber, watching the progress bar creep forward. On the desk, my cup of clove-scented tea had gone completely cold. Through the monitor, Karan’s daughter shifted, her breathing settling into a quiet, rhythmic sigh.

On the terminal, the numbers barely budged:

3488/2184560 MB (0%)

---

#writon