The Hidden Compatibility Issue in Shared Caches: A Lesson for North East India's Tech Industry

The Production Deployment Disaster

In the fast-paced world of software development, rolling deployments have become a common practice to minimize downtime during service updates. However, a seemingly minor change in a model class can lead to significant issues when dealing with shared caches. This article shares a real-life incident that occurred during a production deployment and the solution that made rolling deployments less nerve-wracking again.

The Problem: A Backward-Incompatible Change

During a routine production deployment, the team made a seemingly innocuous change: renaming a single field in a model class. This change, however, introduced a backward-incompatible change that went unnoticed until the deployment started failing partway through.

The Cause: The Shared Cache

The service in question was a data aggregator that consumed upstream data, enriched it, and published results downstream. It didn't own persistent storage but relied on a shared cache for temporary storage. When two versions of the service were running simultaneously, new instances couldn't deserialize cached data written by old instances, and old instances failed on data written by the new ones.

When Does This Problem Occur?

This issue doesn't apply to every architecture. It arises only when all of the following conditions are met: the cache is shared or distributed (e.g., Redis, Memcached), multiple service versions run simultaneously, and those versions read and write the same cache entries.

The Solution: Version Your Cache Keys

Instead of using cache keys like "user:123", the team added an explicit version prefix, such as "v1:user:123" or "v2:user:123". Each service version reads and writes only the keys that match its own version, ensuring compatibility between old and new service versions during a rolling deployment.

Relevance to North East India and the Wider Indian Context

The software development landscape in North East India is growing rapidly, with numerous startups and tech companies emerging. This incident serves as a reminder for developers and teams to be mindful of the often-overlooked cache layer during rolling deployments, ensuring a smoother and more reliable development process.

Looking Forward: A Safer and Simpler Rolling Deployment Process

By versioning cache keys, teams can make rolling deployments less error-prone and more predictable as systems evolve. This small change can have a significant impact on the reliability of software services, making deployments less of a headache and more of a routine maintenance task.