Kubernetes controllers: designing idempotent reconciliation

Start with desired state

A self-service environment should describe what a team needs, rather than a sequence of manual operations. For example, an environment request might identify an application and its access policy. The platform can translate that intent into Kubernetes resources.

A custom resource defines an API for that intent. A controller supplies the behavior. Kubernetes controllers repeatedly compare the requested state with the observed state and work to bring them closer together. Creating the custom resource alone does not implement provisioning.

Make repeated reconciliation safe

Idempotency means that applying the same intent again does not keep producing additional effects. A controller should not create a new namespace or duplicate an external resource whenever it processes the same request.

Use stable resource identities and inspect existing state before deciding what needs to change. Treat partial completion as an expected situation. If namespace creation succeeds but the next operation fails, a later reconciliation should continue from observed state rather than assume a clean start.

Give users an observable outcome

A successful API submission is not the same as a ready environment. Keep user intent in spec and report the observed outcome in status. Status should help a team distinguish an operation in progress from a resource waiting for intervention.

For a platform API, I recommend defining the meaning of readiness before writing the happy path. Decide which dependencies must be available and which failure details are useful to the requester. Avoid exposing credentials in diagnostic messages.

Design deletion alongside creation

Some resources require cleanup before deletion can finish. Finalizers let a controller perform that cleanup while Kubernetes holds the object in a terminating state. A finalizer must eventually be removed by the responsible controller; otherwise deletion remains blocked.

Make the cleanup retryable too. Document what the platform owns, what it retains, and what an operator should inspect if cleanup cannot complete. Treat removal of access as part of the lifecycle, rather than an afterthought.

Test recovery, not only creation

For a provisioning controller, I would start a review with the following scenarios. These are design checks, not a substitute for integration tests against the APIs your controller actually uses.

Writing independent test expectations

• Reconcile an unchanged request twice and check that resources are not duplicated.
• Interrupt provisioning after one successful operation, then retry.
• Change the desired configuration and verify convergence.
• Delete a request and verify cleanup can recover from a temporary failure.
• Check that status describes the outcome without leaking sensitive data.