The Replication of Stone

I. The Tosha-Khana

The air in the old tosha-khana smells of wet lime plaster and scorched dust. For three centuries, this windowless chamber in the eastern wing of the haveli held brass chests of land deeds and copper-plate grants. Today, it houses a single black half-rack containing two refurbished Xeon servers. A noisy desert cooler rattles in the archway, blowing damp, tepid air directly onto the intake fans. It is an operational nightmare, but the thick stone walls keep the temperature from spiking when the Rajasthan summer touches forty-five degrees.

Devendra sat on a low wooden stool, his fingers hovering over a grease-stained mechanical keyboard. He wears a heavy silver amulet of Lord Tejaji around his neck, which clinks against his collarbone whenever he nods. Devendra is twenty-eight, holds a degree from an engineering college in Sikar, and refuses to use modern cloud providers. "Why pay some corporate entity in Virginia to host the reservation ledger of our own ancestors' home?" he had argued when we built the system.

He had a point then. But at 3:15 AM, with the Jaipur Literature Festival only two weeks away and every room from the Jharokha suite to the old stables booked solid, the primary database server, jaipur-primary, was dying.

"The disk is at ninety-two percent," Devendra said, his voice flat with the specific exhaustion of the graveyard shift. He clicked his tongue—a sharp, rhythmic sound he makes whenever a query takes longer than fifty milliseconds. "Four-terabyte NVMe. We are losing forty gigabytes of storage an hour. It is the write-ahead logs. They are not being recycled."

II. The Broken Chain

Our reservation engine, a bespoke system we call ShahiBook, runs on PostgreSQL 15. To comply with local tourism laws and our own paranoia, we replicate every transaction in near-real-time to a standby virtual machine in a Mumbai datacenter.

I looked over his shoulder at the terminal.

SELECT slot_name, active, restart_lsn, pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS bytes_behind
FROM pg_replication_slots;

The output was clean and brutal:

-[ RECORD 1 ]----+---------------------
slot_name        | standby_mumbai_slot
active           | f
restart_lsn      | 7F/1A2C3000
bytes_behind     | 1979120818432

Nearly two terabytes behind.

Three days ago, a JCB excavator trenching for water lines near Ahmedabad had severed the primary fiber link. Our Mumbai standby went dark. Because I had ignored Devendra's warning during the Diwali rush and left maxslotwalkeepsize unset on the primary server—preferring to get a head start on my drive back to Delhi—jaipur-primary did exactly what it was designed to do: it hoarded every transaction log segment since the disconnection, waiting patiently for the Mumbai standby to return and consume them. It was protecting the integrity of our remote replica at the cost of our local survival.

"If the primary disk hits one hundred percent, Postgres will crash on a panic," Devendra said. He reached into his pocket for a bidi, remembered where we were, and tucked it behind his ear. "The booking engine will go down. Guests arriving from Delhi will find no records at the front desk. We have to drop the replication slot."

"If we drop it," I said, "we lose the ability to catch up. The Mumbai replica becomes junk. We will have to rebuild it from scratch."

"Let pg_wal fill and take the primary down?" Devendra turned to face me. His eyes were bloodshot. "Or drop the slot, free the space, and deal with Mumbai tomorrow?"

It was a defensible trade-off, the kind they do not teach in bootcamps. We chose the living guest over the distant backup.

III. The Drop

Devendra cleared his throat, spat out a piece of loose tobacco, and ran the command:

SELECT pg_drop_replication_slot('standby_mumbai_slot');

For thirty seconds, nothing happened. The disk usage remained at ninety-two percent. In PostgreSQL, dropping a replication slot does not instantly purge the files; it merely releases the lock on them. We had to wait for the next checkpoint to trigger database-level garbage collection.

I watched the terminal. Devendra forced a checkpoint manually:

CHECKPOINT;

The server hummed, a low-frequency vibration that seemed to travel through the ancient stone floor. Then, the disk utility output refreshed.

$ df -h /var/lib/postgresql
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1  3.7T  1.8T  1.7T  52% /var/lib/postgresql

Exactly 1.8 terabytes of accumulated write-ahead logs evaporated, recycled by the engine. The primary server breathed again. The immediate danger had passed, but our redundancy was gone. The Mumbai server was now an orphan, its continuous replication chain severed forever.

IV. The Long WAN

Rebuilding required SSHing into the remote Mumbai node, stopping the database service, wiping the local data directory, and pulling a fresh copy from Jaipur over a thin, residential-grade broadband connection that we routed through a local wireless tower.

We sat together as the sun began to yellow the tops of the Aravali hills outside. The peacocks in the courtyard below began their dry, raucous morning calls.

Devendra logged into the remote machine and initiated the rebuild:

ssh administrator@mumbai-replica-01.internal \
  "sudo systemctl stop postgresql && \
   sudo rm -rf /var/lib/postgresql/15/main/* && \
   sudo -u postgres pg_basebackup -h jaipur-primary.internal \
     -D /var/lib/postgresql/15/main -U replicator -P -X stream \
     -C --slot=standby_mumbai_slot -R"

The flag -C was mandatory now; because we had dropped the slot on the primary, we had to instruct pg_basebackup to recreate it before initiating the stream. The -R flag would generate the necessary standby.signal file to keep the remote instance in standby mode.

On the screen, the transfer utility began its slow crawl:

38120/1881290 MB (2%) done

At this rate, the backup would take eighteen hours.

Devendra stood up, stretched his arms until his joints popped, and walked out to the terrace to finally light his bidi. I remained in the damp room, watching the progress bar tick from two percent to three. Outside, the desert cooler rattled against the stone archway, blowing another gust of humid, warm air across my face.

---

#writon