Silent Data Corruption in Distributed Systems: A Case Study from PagerDuty
In the digital age, ensuring data integrity is paramount for any organization, especially those operating distributed systems. A recent incident at PagerDuty, a leading digital operations management platform, serves as a stark reminder of the hidden dangers that lurk within complex systems.
The Night That Changed Everything
At 3:00 AM on a Tuesday, an unexpected alert from PagerDuty signaled the start of an unusual journey for the PagerDuty team. The system, built using Spring Boot, PostgreSQL, and the Saga Pattern, was functioning normally, with no visible errors. However, something felt off, leading to a postmortem investigation that uncovered a silent data corruption.
The Architecture and the Saga Flow
PagerDuty's system uses an orchestration-based Saga, owned by the HR service. The goal is to create Auth credentials and an HR profile for a new employee, with both operations needing to succeed or neither should exist.
The Trigger: A Harmless Enum
The introduction of a new enum value, 'WORK', seemed innocuous at first. However, it triggered a chain of events that led to data corruption.
The Real Culprit: Schema Drift
The HR database had a PostgreSQL CHECK constraint that only allowed certain values for the 'address_type' field. When 'WORK' was saved, it violated this constraint, but the Saga failed to roll back, leading to ghost users in the system.
Lessons Learned and Recommendations
Fail Fast with flush()
The initial Saga design led to deferred database operations, which caused the error to go unnoticed. By forcing SQL execution immediately upon saving the employee profile, errors surfaced immediately, allowing the Saga to respond appropriately.
Repair the Schema
A minor alteration in the HR database schema resolved the check constraint issue, allowing 'WORK' to be saved as a valid 'address_type' value.
State-Based Saga
The traditional delete-based rollback approach proved fragile in this case. Switching to a state-based saga, where only state transitions are made, provides a more robust solution.
Reconciliation Job
A scheduled job that checks for and cleans up orphaned users has proven to be a lifesaver in similar situations, preventing data corruption from accumulating over time.
Implications for North East India and Beyond
As organizations in North East India and across India adopt more complex distributed systems, understanding the hidden dangers and taking proactive measures to ensure data integrity is crucial. The PagerDuty case study serves as a valuable lesson for all.
Reflections and Looking Forward
The PagerDuty incident underscores the importance of vigilance and continuous testing in distributed systems. While technology advances at a rapid pace, the human element remains a critical factor in maintaining system integrity. As we move forward, let us remember to question, test, and adapt to ensure our systems remain robust and reliable.