The Illusion of Cache Consistency: Surviving Stampedes and Zombie Keys

The 3:14 AM Cascade

At 03:14:22 UTC, replica node db-prod-04 logged a transient 40-millisecond lock contention on the user profiles cluster. By 03:14:26, the cache hit ratio plummeted from 99.2% to 11.4%. The Redis cluster was entirely healthy, memory utilization sat at 38%, and yet the primary Postgres pool exhausted all 2,000 pooled connections in four seconds.

Every engineer knows the Phil Karlton adage about cache invalidation and naming things. Yet in production environments, the most lethal cache bugs rarely stem from calculating wrong keys; they emerge from the temporal gap between cache eviction and downstream persistence.

When thousands of concurrent fibers discover a cache miss simultaneously, your cache stops being a shield and becomes an accelerant for database collapse.

Anatomy of a Cache Stampede

Consider the naive cache-aside implementation common in microservices:

func GetUserProfile(ctx context.Context, userID string) (*UserProfile, error) {
    key := fmt.Sprintf("user:%s", userID)
    
    // Attempt cache read
    if cached, err := redisClient.Get(ctx, key).Bytes(); err == nil {
        var profile UserProfile
        if err := json.Unmarshal(cached, &profile); err == nil {
            return &profile, nil
        }
    }

    // Cache miss: hit Postgres
    profile, err := db.QueryProfile(ctx, userID)
    if err != nil {
        return nil, err
    }

    // Populate cache asynchronously or inline
    data, _ := json.Marshal(profile)
    redisClient.Set(ctx, key, data, 10*time.Minute)

    return profile, nil
}

Under steady traffic of 15,000 QPS, evicting user:84920 (e.g., after an account update) triggers a classic thundering herd. Between the moment Redis deletes the key and Postgres finishes executing SELECT * FROM profiles WHERE id = 84920, hundreds of concurrent read requests encounter an empty cache. Every single goroutine bypasses the cache and pounds the database with duplicate queries.

Mitigating the Stampede with Singleflight

To eliminate redundant fetch executions, distributed applications should collapse duplicate in-flight requests into a single caller using synchronization primitives like Go's golang.org/x/sync/singleflight:

var requestGroup singleflight.Group

func GetUserProfileSafe(ctx context.Context, userID string) (*UserProfile, error) {
    key := fmt.Sprintf("user:%s", userID)

    if val, err := redisClient.Get(ctx, key).Result(); err == nil {
        var p UserProfile
        if json.Unmarshal([]byte(val), &p) == nil {
            return &p, nil
        }
    }

    // Suppress concurrent duplicate executions across local goroutines
    res, err, _ := requestGroup.Do(key, func() (interface{}, error) {
        profile, err := db.QueryProfile(ctx, userID)
        if err != nil {
            return nil, err
        }
        
        data, _ := json.Marshal(profile)
        // Set with jitter to prevent synchronized expiry cascades
        jitter := time.Duration(rand.Intn(60)) * time.Second
        redisClient.Set(ctx, key, data, 10*time.Minute+jitter)
        return profile, nil
    })

    if err != nil {
        return nil, err
    }
    return res.(*UserProfile), nil
}

Leases and Early Eviction Probabilities

Singleflight protects single node process boundaries, but across a horizontally autoscaled fleet of 80 stateless API pods, up to 80 simultaneous database queries can still strike Postgres per cold key.

To solve fleet-wide stampedes, adopt probabilistic early expiration (XFetch algorithm) or cache leases:

1. Deterministic Leases: When a cache miss occurs, Redis returns a short 10-second token lock via SET user:key:lock token NX EX 10. Only the token holder is authorized to query Postgres and re-hydrate the cache. Non-lease holders wait 50ms or read a stale version.
2. Probabilistic Refresh (XFetch): Compute refresh likelihood on each read. If the threshold is reached, a background worker proactively refreshes the key before TTL drops to zero, ensuring user-facing reads never experience a synchronous miss.

Architecting resilient distributed caches requires treating cache reads as probabilistic state machines rather than deterministic key-value stores.