EXPEDITION 09 · 2024
Infrastructure
Drift Watch
A configuration drift detector that produced more noise than useful operational evidence.
73%low-value alerts
2 wkuntil trust declined
2024year explored
The question
Could every unexpected infrastructure change become immediately visible?
The route
I compared snapshots too literally and underestimated legitimate runtime mutation across controllers.
The outcome
Alert fatigue set in quickly, and operators stopped trusting findings that lacked ownership and severity.
Field note
Detection without a response contract simply converts uncertainty into notification volume.
Instruments
- Kubernetes
- Go
- Prometheus
- Git