Description
Every composite on both estates running this platform reports Responsive=False with reason WatchCircuitOpen. This is Crossplane’s own protection working, not a Crossplane bug: the composite controller wraps its composed-resource watches in a token bucket breaker whose defaults allow a burst of 50 updates and one update every 2 seconds sustained, stays open for 5 minutes, and lets a probe through every 30 seconds. Something is updating composed resources fast enough to exhaust that budget continuously.
Observed live on 2026-09-16. Objects are rewritten every few seconds with byte-identical content: two snapshots 25 seconds apart differed only in resourceVersion, with spec, status, conditions and managedFields all unchanged, and metadata.generation was stable, so this is not the composition re-applying a changed spec.
The affected kinds are specific and the pattern is worth starting from. AccessPolicy churns on both estates. On the estate that also vends in-stack content, FolderPermission and DashboardPermission churn by roughly seven times as much as anything else, followed by Folder and Dashboard. Stack, StackServiceAccount, Team, Role, RoleAssignment, RoleAssignmentItem, OrganizationPreferences and PluginInstallation never trigger the breaker at all. The kinds that churn are the ones carrying unordered list fields (access policy scopes, permission lists) and the ones that do not carry such fields stay quiet, which points at list-ordering instability between what the composition sets and what the API returns, producing a diff on every observe. That is a hypothesis from the correlation, not a proven cause.
Impact today looks bounded: event-driven composite reconciles are throttled to one probe per 30 seconds while the breaker is open, and poll-driven reconciles continue, so nothing is stuck. The costs are slower convergence after a real change, continuous API churn against the vendor, and a Responsive=False condition on every composite, which makes a genuinely degraded composite indistinguishable from the normal state.
Acceptance Criteria
- #1 The source of the repeated no-op updates is identified per affected kind, with evidence showing what the provider sends and what the API returns
- #2 Whether the churn originates in the provider or in what this composition sets is settled, and the finding names which of the two has to change
- #3 Either the churn is removed for the kinds this platform composes, or the platform records why it is acceptable, what it costs and what it would take to fix
- #4 A composite with no failing children reports Responsive=True, or the condition is documented as uninformative with the reason
Definition of Done
- #1 just check passes locally
- #2 hosted Validate workflow passes on the completing commit
Implementation Plan
Wave 13: derive candidate churn fields from the renderer and pinned CRD without live contact, test-first the permission-kind correction if evidenced, and return any AccessPolicy edit as a wiring packet.
Implementation Notes
2026-09-18 live re-verification against both estates. Two claims in the description are WRONG and one is confirmed. Read this before starting.
CONFIRMED: every composite on both estates still reports Responsive=False with reason WatchCircuitOpen, and the affected kinds are exactly the three named. Evidence is metadata.generation, which increments only on a SPEC write, so it counts spec rewrites directly and is a far better instrument than sampling resourceVersion.
Estate with in-stack content, one object of each kind: DashboardPermission generation 518499 FolderPermission generation 517384 AccessPolicy generation 202187 Folder 2, Dashboard 3, Team 2 and 3, OrganizationPreferences 2, Stack 3
Estate without in-stack content, seven access policies and two stacks: AccessPolicy generation 191, 191, 191, 79963, 81197, 82591, 85959 Stack generation 3
So the quiet kinds sit at 2 or 3 generations and the churning kinds sit between 80 thousand and half a million. The ratio is roughly 170000 to 1, and the permission kinds do churn about 2.6 times as much as AccessPolicy, which the description’s ‘seven times’ overstates but gets the direction right.
WRONG 1: ‘metadata.generation was stable, so this is not the composition re-applying a changed spec.’ Generation is not stable, it is enormous. The 25-second window it was sampled in was too short, and at least one object is quiet for 20 seconds at a time, so the churn is bursty per object rather than continuous. Something IS rewriting the spec, tens of thousands of times. A composed resource’s spec is written by the composite controller from what the composition function renders, so the description’s own AC2 question now leans hard toward this repository rather than the provider - but which field is still open.
WRONG 2: the list-ordering hypothesis is FALSIFIED for AccessPolicy. Scope ordering does not correlate with churn, and it runs the wrong way: generation 85959 - spec scopes NOT sorted, atProvider sorted generation 82591 - spec scopes ALREADY sorted, atProvider identical generation 81197 - spec scopes NOT sorted generation 79963 - spec scopes ALREADY sorted, atProvider identical generation 191 - spec scopes NOT sorted Two of the four highest-churn policies already emit their scopes in exactly the order the API returns, and the lowest-churn object emits them unsorted. Sorting the scopes list will therefore not fix this. Do not start there, and do not re-derive the hypothesis: it has been tested and it is dead.
THE DISCRIMINATOR TO START FROM instead. On the estate with no in-stack content, four AccessPolicy objects churn and three do not, same kind, same cluster, same provider. The three quiet ones sit at exactly 191 generations each, which is suspiciously equal and suggests they stopped rather than never started. Diff a churning object against a quiet one field by field - spec, status.atProvider, managedFields ownership - and the differing field is the cause. That comparison is cheap and it is the whole job.
One more structural clue worth testing: all three churning kinds are WHOLE-SET replace resources, the ones the wave operating model already singles out as needing exactly one declarative owner. The quiet kinds are not. Two writers fighting over a whole-set field would produce exactly this signature.
Also observed, and separately relevant to GCV-0075 AC4: AccessPolicyRotatingToken exposes NO expiration, secondsToLive or earlyRotationWindowSeconds in status.atProvider at all - all three read null - while StackServiceAccountRotatingToken exposes every one of them. An operator cannot see when an access policy token expires from the resource.
Wave 13 terminal reconciliation: pinned-source evidence identifies resolver and late-initializer scalar writes as the no-op spec-write source for AccessPolicy, FolderPermission, and DashboardPermission. Focused tests prove the renderer preserves observed identifiers, UIDs, team IDs, and org IDs. No live cluster contact was permitted, so generation settling and Responsive=True remain unverified.
Loop 18 current function pin is 886887d0315a2a6ae5182732f184cc873a5d1fa2 (sha256:c2e55953a4ec28376aed668d5d58c80932f668e24e5242699f595d2d1c282d08). AC4 remains estate rollout and multi-window Responsive=True observation.
Loop 22 pin-roll boundary: signed function package from successful main publish run 36766188067 resolves to sha256:b227db268335776e050c81548818aa532d71f82992d4f84a12a95ac3ee70ef3f. Install references and installation row move together; estate rollout remains owner action.
Loop 22 final estate pin-roll boundary is signed function digest sha256:29ca2da4146145e330d5c01c3e63f5beba18f15d38b7526ff3b51ef3ca1d9266, published from main17bc31ad2d63c9e7e7e257c6c2466bb8874e116e by run36788719877. Estate rollout remains owner action.
Loop 22 final pin-roll boundary now includes the preserved automatic protobuf update: signed function digest sha256:44273e53f0e7c77264e631c9139841c23eae42943bacbff23c7176ac509a7fd0, publish36791008118 from main8361eda2dad9c7066c25cc5e556c619b81e75853. Owner estate rollout remains open.
Loop23 estate A pass1 live readback at 2026-10-02T17:55Z: both stack composites report current-generation Synced=True, Ready=True and Responsive=True with WatchCircuitClosed since 17:29:16Z. Managed-resource census remains twenty Synced and twenty Ready, unchanged from pre-state. Platform carries the released controller and function digest; both fresh signature Jobs were observed Complete. AC4 live observation is now available; final task completion still needs the completing local/hosted gate identity.
Loop23 AC4 live verification only: no new source implementation or review-repair attempt; prior source attempt lineage unchanged, infrastructure retries 0, no new grant. Both observed composites have no failing managed children and Responsive=True with WatchCircuitClosed.
Loop23 function pin tail is the signed package sha256:1968d0f84fa8926fe762ae2976eccc2625770e5bef412cd9d879aeab105cfe5a from Publish 36899690651 on 6a28015eaa96a5b3aafc343b7d786ee37af0a43b. Existing AC4 live signal was proven on the prior released digest; the final estate release repin will receive this completing function package.
Final Summary
Parked after landing the source-backed renderer correction. The composition now preserves provider-resolved fields for all three affected kinds, with focused tests and integrated just check plus hosted Validate run 35333659147 green at b47831ddddfd5ec10e1e699d4d8608886261becd. Resume after deploying this pin: observe generations over several burst windows and confirm a healthy composite returns Responsive=True. AC4 remains unproven.
AC4 completed by estate A pass1 post-state at 2026-10-02T17:55Z: both composites are current-generation Synced=True, Ready=True and Responsive=True; all twenty managed resources remain Synced and Ready. Completing repository SHA b799100cf9ad691a08a18c259ad732029d191f86 passed the full local gate and hosted Validate 37047155497. The released function and carried provider are healthy live, with fresh verifier Jobs observed Complete. Existing source corrections now have the required live responsive signal.