Separate real drift from intended change and tool noise
Infrastructure drift occurs when the configuration of a deployed resource no longer matches its IaC declaration or recorded state. Common causes include console changes made during troubleshooting, service-managed attributes, automation outside the main delivery pipeline, and new defaults introduced by a cloud provider. Not every difference is an incident, but every unexplained difference increases deployment risk. A routine module upgrade can unexpectedly remove an emergency fix, replace a resource, or tighten a dependency that operators did not know had changed.
We classify findings into four practical groups. Unauthorized drift should be investigated or reverted. Emergency changes must be reconciled into code and the audit trail. Externally managed fields require an explicit ownership boundary. Non-material noise should be handled through pinned versions, normalization, or narrowly scoped ignore rules. Adding ignore_changes whenever a plan is inconvenient is dangerous: ignoring a whole resource may hide meaningful changes to networking, encryption, retention, or access control.
Build a repeatable, read-only detection pipeline
Drift detection should run from a controlled environment rather than an engineer's laptop. For Terraform, that typically means initializing with locked CLI, provider, and module versions, then producing a refresh-only plan or a normal plan against the intended workspace. Platforms such as CloudFormation can also use their native drift facilities. Native checks are useful but may not cover every resource type or every property, so teams should document those gaps instead of assuming that a clean report proves complete alignment.
The detection identity should have read-only permissions wherever possible. Remote state needs encryption, locking, access controls, and recoverable version history. Schedule frequency should reflect impact: identity policies, internet-facing rules, keys, and production databases deserve faster detection than disposable development resources. Each run should retain the timestamp, commit SHA, account and workspace, tool versions, and a concise difference summary. Full plans must be treated as controlled artifacts because providers can expose identifiers or sensitive values in output.
- Establish a baseline: confirm that the current environment can be explained by the repository, and resolve longstanding unknown differences.
- Reduce noise: pin versions and ignore only attributes that are verified to be owned by a service or another controller.
- Route by risk: escalate privilege expansion, public exposure, deletion, and forced replacement; send cosmetic metadata changes to a normal queue.
- Keep detection non-mutating: scheduled jobs should report rather than apply, except for narrowly bounded and thoroughly tested remediation.
Review operational impact, not just configuration syntax
An IaC pull request should include both the source diff and a plan generated for the target environment. Reviewers should first verify that the plan belongs to the latest commit, correct cloud account, region, and workspace. They can then examine creates, in-place updates, deletions, and replacements. Reviewing HCL, YAML, or module inputs alone is insufficient because provider behavior, module internals, and existing state determine what the platform will actually do.
The most useful review questions are operational. Does the change broaden an IAM permission or network source range? Could it replace a database, disk, queue, load balancer, or address? Is the proposed order safe for dependent applications? Does the deployment require a backup, staged rollout, maintenance window, or tested rollback procedure? For shared modules, reviewers should also identify every consuming environment and avoid approving a broad version bump based on a plan from only one workspace.
Policy as code is valuable for objective guardrails such as requiring encryption, blocking unrestricted administrative ports, or enforcing approved regions. It should not be expected to understand every service dependency or data-lifecycle decision. Human approval is still important for context, but large raw plans make weak review material. Generate a readable summary, emphasize destructive and security-sensitive actions, collapse unchanged data, and provide the full artifact for deeper inspection. Never publish a plan containing secrets into an unrestricted pull-request log.
Give emergency changes and remediation a controlled path
A total ban on console changes is often unrealistic during outages or access failures. A better control is a break-glass process with time-bound elevated access, recorded operator identity and reason, a constrained scope, and an immediate follow-up item. After service is stable, do not blindly run apply. First decide whether the repository should adopt the live configuration, whether an existing resource must be imported, or whether the environment should return to the declared state.
The correct remediation depends on which side represents the intended design. If an emergency adjustment proved necessary, update the IaC first and verify that the resulting plan contains only the expected reconciliation. If a change was unauthorized and reversal is safe, restore it through the normal reviewed deployment path. When a difference could delete data or recreate a critical resource, pause automation, confirm backups and dependencies, and schedule the work explicitly. Mature IaC practice is not defined by permanently reporting zero drift; it is defined by discovering every meaningful difference, assigning ownership, reviewing its impact, and converging safely. In integrated cloud and enterprise environments, the platform and application teams should establish those boundaries together.