Backlogsm7kni.io

Projects / grafana-cloud-org-insights / GCI-0119.15

Task · GCI-0119.15

Prove the labelling pillar live on the dev deployment

Description

Prove the released labelling pillar on the full discovered dev estate with one fresh successful scan, manual or scheduled. Record measured runtime, actual task sampled memory and bounded known-value absence from the private label inventory and its published surfaces. Preserve honest unknown/partial coverage and live dashboard evidence. The owner withdrew the multi-day score-stability requirement on 2026-10-14; historical test-process RSS is not a deployed collector memory baseline.

Acceptance Criteria

Definition of Done

Implementation Plan

loop14: root-only dev2 to signed0.8.0 after account-wide original customer mutator quiescence and restored AWS auth. Current auth hold; no dev apply/start or label customer enablement. AC1 runtime/rawabsence/measured collector memory qualified now; two-day natural stability evidence-later. 232.7MiB historical pytest RSS is NOT collector/Fargate baseline; task reservation NOT usage. Require saved targeted plan classification/exactapply, privacy/reader/identity readbacks, natural fullT2 and actual dashboard rendering; no substituted test RSS or ideal evidence.

loop16 D-ORDER16: signed v0.8.1 dev-first saved-plan apply, label-inventory on, measured T2 with Container Insights enabled only for that window then disabled, per-stack p95 and sink privacy evidence, live dashboard rendering. Two-day stability remains verify:2026-10-10. No source repair allowance renewed.

loop18: accepted signed release0.8.4source6a689ecc; exact dev pin upgrade, security review, committed consumer rebuild, saved targeted plan and exactapply; D-MEM18 measuredT2 with first-failure AWS auth parking, raw absence and live render; nochange plan. Stability evidence-later only after current proof passes.

loop25 D-NAT25: bounded authentic natural-capture adapter reviewed field-by-field against unchanged verifier, dev-only CI enable/readback and schedule/label config readback then process start until 2026-10-15T08:00Z. No auth login or synthetic/backfilled evidence.

2026-10-13 direct-main follow-through: preserve loop25 failure artifacts; establish a fresh bounded capture with the existing verifier and owner-selected score tolerance. Revalidate runtime identity and startup, capture three future daily runs over at least 48 hours, retain first-auth-failure stop, and disable dev Container Insights with fresh identity/readback after the final capture. No customer mutation or source-floor change.

Implementation Notes

AC3 proved: publicv0.8.0 sourcebe4d0de, GHCRbf382117 twoarches sourcebound/pinnedcosign, releaseplease37602451585 nestedstablepublish leaves success and exactCI37602450367requiredleafsuccess. Edge/helm skips separate. AC1/2unproven; original customer mutator outcomeUNKNOWN blocks live account serialresource and AWSauthexpired09:33, firstsingleSTS10:03:06.

loop14terminal: AWSauth expired09:33; exactlyoneSTS each10:03:14,11:03:20,13:03:26 failed255, no login/repair. Original customer provisioner terminalUNKNOWN holds accountserialmutex; no dev2write/newtask. AC3signed0.8.0Done, AC1runtime/measuredmemory/rawabsence andAC2rendering unproven; stabilityevidence-later. Resumeonlyafterauth+exactrecordedmutatorquiescence, then saveddev2plan/identity/privacy/readers, naturalT2/memory/rawabsence/rendering.

loop15: GCI-0119.15 (prove labelling live on dev) remains dependency-parked, implementation attempts 0 in this loop. Signed v0.8.1 source b8e7ee5a3ae21e9bd7187c74e0a3690ec0b463ef/image ca873b4b8fbdfd1e101c7c87873d32f166f09e1ebe6827f99147276e7243aeeb accepted. AWS auth recovered, but required prior customer acceptance failed its full T2 publication floor; see GCI-0125 (resolve Synthetic owner-input coverage before deployment acceptance). No dev apply, label enablement, Container Insights window, runtime/memory/raw-absence or live labelling UI witness occurred. Existing source attempt allowance remains exhausted; do not infer failed live AC from an unattempted deployment. Resume only after the customer prerequisite is accepted or the owner explicitly re-grades the order.

loop16: GCI-0119.15 (prove labelling live on dev) unaccepted and dependency-parked, implementation/live-proof attempts1, review-repair rounds1 for observational wrapper only, source07 allowance not renewed. Signed genericv0.8.1 verified; consumer image/source/committed overlay bound and exact saved dev plan applied. Measured deployed T2 STOPPED exit1, descriptor captured1.55s after observer. label_inventory owner input17/20=85% below90% floor, errors0; all publications correctly refused and last-good T2 unchanged. Five-stack paired label_risk+label_inventory p95 nearest-rank33.468s, wrapperwall144.303s vs3600s. Container Insights measured123MiB deployed1-minute sampled peak, tasklimit1024MiB, not comparable to historical232.7MiB pytestRSS. Dev-only CI enabled/read back then disabled/read back. Raw absence on new outputs and live labelling rendering remain unproven, no stability verify label/Done claim. Resume after unavailable-signal disposition or new source07 allowance; no floor lowering or silent source repair.

loop17: no new live-proof attempt (historical loop16 live attempt1 persists); config/build preparation1 only, no apply/task. Signed R-rel5v0.8.3 source083df8b/GHCR89ff83fe accepted; dev-only desired module/manifest labelenablement1 committed/pushed infrastructure8aa197, preserving foreignEKSupdate, declarationsecurityPASS, localvalidate0 and exactCI37702640669success. Localconsumerb588f5bc binds source083df8b/infra8aa197/overlay27a9d085; NOECRpush or imagepin yet. AWShold3probesfailed255; no reader/privacy snapshot completion, savedplan/apply, D-MEM17window, measuredT2, newoutputrawabsence or live rendering. LastobservedliveR-rel4consumerad48 labelOFF. AC1/2OPEN; no stabilityDone/verifylabel. Resume freshSTS, currentrealstate baselines, ECRpromotion and reviewedexactplan/apply, measuredT2 withCIenabledonlywindowthenoff, runtime/privacy/render and finalnochangeplan. Desiredlocalconfig is NOTdeployed.

loop18: retained signed v0.8.4source6a689ecc, consumer OCIindexa27ba7d1 builtfromcommitteddf782dc8, exact reviewedplan654afb04 applied atinfra60650a8. All5taskdefs/projections/schedules and effective schedulerpolicy equalapprovedplan; existing5basicNone readers/IDs/pairs/tokens/SSMversions unchanged, privacycontrols freshPASS. ONE measureddevT2 exactARNending029701716d93400e9c0336536d6baff5 STOPPEDexit1, descriptorcaptured12.53s afterstop. Missingreader on sixthfreshstack caused primarysource5/6 refusal; no T2publication and lastgoodunchanged. Labelinventory20/24passes0.80:metrics6/6,logs3/6(missing_input3),traces5/6(missing_input1),profiles6/6. Per-stack combinedlabel-risk+inventory p95nearest-rank33.914s over6; wrapperwall120.040s vs3600s. ContainerInsights92MiB deployed1-minute sampledmax/3samples versus1GiBlimit; samplewindowNOTfulltaskcoverage within60s, not strictpeak/RSSbound and not comparable to historical pytestRSS. CIenabled/readback then disabled/readback. Cumulative liveproofattempts2 (loop16one,loop18one), thisloophelperreview-repairrounds1; source implementation allowance unchanged/exhausted, no reset. AC1/2 remainOPEN: raw-value-absence on newpublishedoutputs, live rendering and two-daystability unproven; no verify label or Done claim. Fixforward via GCI-0119.15.01 (restore newly discovered dev reader coverage); no rollback, credentialmint, scope/floor/optout change or replacementtask.

loop18finalconfigurationgate: nochange targetedplan bcb12cd2fc0542189520e2132ebc0577344c8fba2079725ec04532a45e37ade7 atinfra60650a8 has0resource/0output non-noop actions. Deployedconsumer retained, no rollback. This proves configuration convergence, not successful T2 publication or labelling acceptance.

loop19 (main thread): v0.9.0 dev T2 (ARN ending bfd739f4f3db46ee8fb4c39575b86080) exit 0 but labelling partial 19/24=0.79: root cause a source defect, Loki /labels and /label//values return exact 200 {“status”:“success”} with no data key on empty tenants (3 dev stacks), classified missing_input. Fixed forward in 7b72bb5, released v0.9.1 45621a9 (cosign pinned identity/issuer/repo/source verified). R-dev8 on v0.9.1: consumer sha256:b2277d28…7ffa, saved targeted plan classified (image + schedule targets only), exact apply, post-apply readback PASS. Measured T2 ARN ending b9191dd383524a6dbc66a39fb13b3964: exit 0, descriptor lag <1 s, primary sources 6/6, labelling 23/24=0.958 (metrics 6/6, logs 6/6, traces 5/6 missing_input 1, profiles 6/6), per-stack p95 26.65 s, wrapper wall 252.5 s vs 3600 s, memory 127 MiB deployed 1-minute sampled peak (Container Insights enabled for the window only, disabled and read back). Raw-value absence PASS (bounded sample: hydration, four labelling views, metric series, Loki task window, stdout); bucket privacy PASS before/after. AC #2: labelling dashboard published to robk and rendered by Grafana headless renderer with OCR: title seen, 10 numeric regions, no render error, at least one No data panel present (not per-tab grading); 8 project alert rules kept their observed paused/unrouted state. Two-day stability is evidence-later verify:2026-10-11. Dev no-change plan NOT yet run: AWS SSO expired at that step (D-AUTH19 park). Evidence /Users/rob/repos/grafana-cloud-org-insights/codex/loop19-evidence/R-dev8/.

loop23: zero implementation attempts; read-only natural-proof capture found five matching T2 launches since 2026-10-09 and healthy published scans. Exact linked ECS descriptor returns MISSING; per-task measured memory windows not captured. just verify-labelling-stability /Users/rob/repos/grafana-cloud-org-insights/codex/loop23-evidence/dev/natural-stability.json exits 1 (unverified). This is historical evidence loss, not source failure. Immutable scans alone do not satisfy full artifact contract. Resume only with complete authentic dated task/memory captures or an owner-graded fresh comparable natural-run campaign. No synthetic witnesses, changed verifier or Done claim.

AC #1 unchecked in loop23 because its combined runtime/memory/raw-absence/two-day-stability criterion is not fully proved: historical loop19 deployed runtime/memory and sink privacy evidence remains valid for that measured run, but natural dated full-artifact stability does not pass. This correction preserves all prior evidence and does not change the verifier or requirement.

loop25 utility final83d8464337a9f3ae49ef2b0841359f20bcf020be5947544e84dd936322ee356f accepted after fail-first synthetic subprocess/verifier proofs, independent field/security deltas and CodeRabbit delta complete0. Helper cycles3, review-repair2; historic deployed/live proof remains separately recorded and source07 allowance unchanged. Root dev CI enabled/read back, current T2 ENABLED labelinventory1/image/projection unchanged. Exact launchd job gui/503/com.rknightion.loop25.natural-capture ran once PID58799, first STS command exceeded15s and child was killed by read-command bound, stderrempty: underlying causeUNKNOWN, not diagnosed credential expiry. Process stopped exit3 per D-AUTH18, no authenticated batch or observations, no login/relaunch. CI remainsENABLED, owner open item GCI-0119.15.02 (disable dev Container Insights). Requested verify:2026-10-15 retained as explicit failed/incomplete verification due date, NOT Done or a promised future pass; verify-cmd: just verify-labelling-stability /Users/rob/repos/grafana-cloud-org-insights/codex/loop25-evidence/dev/natural-stability.json. Fixed window alone only2runs24h cannot>=3runs48h. Resume requires owner-graded fresh campaign/environment/bound, no backfill or automatic retry. Evidence /Users/rob/repos/grafana-cloud-org-insights/codex/loop25-evidence/dev/capture-start-readback.json and /Users/rob/repos/grafana-cloud-org-insights/codex/loop25-evidence/dev/capture-start-gap-cause.json.

2026-10-13 direct-main readback: old launchd capture remains stopped. Fresh STS succeeds with dev account and with the previous explicit launchd environment; prior timeout cause remains unknown. Current dev T2 schedule ENABLED at 04:35 UTC, label inventory enabled, image/projection unchanged, zero running tasks. Prepared source only changes fixed end date to 2026-10-17T08:00Z; NOT started. Matching selected-profile SSO access-token cache expires 2026-10-13T16:13:04Z and has no refresh token, so a usable unattended three-day identity is unproven. Owner score tolerance question pending; no auth login, token copy, restart or proof relaxation. Container Insights still enabled.

2026-10-13 continuation: fresh selected-profile STS identity failed exit255 in1.55seconds. Live chain stopped at authentication prerequisite, no login/retry/task/capture restart. Existing prepared capture remains unstarted; Container Insights last observed enabled, not re-read after authentication failure.

Rob selected maximum score movement10points on2026-10-13. Fresh AWS auth succeeds but selected token expires22:47UTC without refresh; three-day unattended identity remains unproved. New comparable natural capture must begin after any requested deployment changes, preserve unchanged verifier and authentic full artifacts.

Finalcomparablecapturefreeze15source d5f559e6ada2629d8f42cbc85edfb7f726f2b9f6, actualdevT2ECRmanifestdigest/runtimeed013/catalogue5/deadline3600/scoretolerance10 boundfromlive15TDandENABLED04:35schedule. DevCIpreservedENABLED. Boundedreadonly naturalcapturePID5621 started2026-10-14~02:58UTC at /Users/rob/repos/grafana-cloud-org-insights/codex/direct-20261013/dev/natural15-capture; authenticatedbatchesobserved, zeroobservationsyet, UNVERIFIED. Fixedbound2026-10-17T08:00UTC, stopsfirstauthfailure/no login/backfill; currentSSOexpires07doesnotprove3dayidentity. Threecomparabledailyruns>=48h/rawabsence/authenticmemory/taskclockand10pointacceptance remainopen; customerlibrary/auditingdependency andCIholdnotwaived.

StatusnowInProgressbecauseboundednaturalcaptureisactivelycollecting, notaccepted. Allrequired48h/3day/10point/memory/raw-absencecriteriaremainuncheckeduntilunchangedverifierpasses. Otherretainedfeaturecontractsandcustomerlibrary/auditholdremainParked.

2026-10-14 owner replaced the three comparable daily runs over 48 hours with one successful fresh dev scan. Manual scans explicitly authorized. Multi-day score stability is withdrawn, not proved; existing publication floors, privacy and measured runtime/memory checks remain. Read-only investigation underway: today natural T2 refused publication on signal_inventory 5/6 with a traces invalid_response; latest labelling input remains dated October 13. Capture stopped with an authentication/authorization gap and zero qualifying observations; not restarted.

2026-10-14 revised single-run acceptance proved on signed v0.13.6 source e7910efdd55738150b0d867dec816132aef468ff. One explicitly manual dev T2 original ARN suffix cf4d7e81712a4fa19a8903d4186cc56f exits0; native collection166.12s, actual ECS lifecycle367.734s, all6/6 primary stacks and all owner sources healthy, signal_inventory6/6. Label source availability23/24=0.9583; incomplete rule coverage/scores remain absent as designed. Exact version-specific S3 body/ETag/version/window bound retained. Five authentic Container Insights samples max152MiB vs actual1024MiB task limit. Execution window endpoint gaps46.225/56.672s; ECS lifecycle last-sample gap81.509s includes post-execution cleanup, so original three-day full-lifecycle tail60s verifier is UNVERIFIED and not claimed passed. Measured execution sample witness is explicit, not a strict lifetime/cgroup peak. Bounded privacy60transient known instance values from3fresh live stacks: no occurrences in private label input, four views, actual26Mimirseries,75nativeLoki lines and131stdout events. No raw samples saved; existing risk-label exception excluded. Live UI AC2 retained from prior actual browser/backend proof, dashboard source unchanged. Required current source gates and exact released CI all pass. Evidence /Users/rob/repos/grafana-cloud-org-insights/codex/direct-20261014/tempo142/manual-dev/; fresh independently read dev inventory/library+retention views match6scannable stacks at /Users/rob/repos/grafana-cloud-org-insights/codex/direct-20261014/tempo142/dev-rollout-prerequisite2/verified.json. Multi-day stability withdrawn by owner, never proved.

View the source file on GitHub