Config drift is a people problem
Every configuration management system eventually grows a second, informal system: the handful of manual changes that nobody wrote down.
The drift itself is rarely interesting. Someone bumped a timeout during an incident, added a feature flag by hand, or changed a replica count because the autoscaler was misbehaving. The interesting part is that the change was reasonable at the time, and then the context disappeared.
Drift is a symptom
When I see a large diff between the declared state and the running state, I do not start by writing a better reconciliation loop. I start by asking why the change was easier to make by hand than through the normal path. Usually the answer is one of three things: the normal path is slow, the normal path is scary, or nobody knows who owns the normal path.
A tool that only reports drift is useful, but it is also a way to feel productive without fixing the reason. The report becomes another dashboard that everyone acknowledges and nobody reads.
Make the desired state cheap to read
The first practical step is to make the declared state something a person can review in a pull request. If rendering the config takes five minutes and three credentials, people will edit the running system instead. A small renderer and a diff in CI are worth more than a weekly audit.
#!/usr/bin/env bash
set -euo pipefail
diff -u <(render prod) <(render staging)The second step is to make the diff visible where the change happens. A bot comment on the pull request is better than a nightly email, because it arrives while the author still remembers why they touched the config.
Ownership beats automation
Every piece of configuration needs a name next to it. Not a team alias, not a Slack channel, but a person who will notice when it changes. That sounds bureaucratic until the first time a stale flag takes down a service and the only answer is “I think that was from the old migration.”
Automation helps, but it cannot decide whether a difference is intentional. A human has to say “this is the new normal” or “this is a mistake.” The smaller the diff and the closer it is to the change, the cheaper that decision becomes.
Finally, delete things. A config file that has been carried through five migrations is not free; it is a set of assumptions that nobody has tested. Removing a dead flag is often the highest-leverage drift fix available.